You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2021/09/01 16:04:08 UTC
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #450
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/450/display/redirect?page=changes>
Changes:
[emilyye] sync nltk, orjson for Python image
[noreply] Allow `google-auth < 3`
[samuelw] [BEAM-12776] Change closing to happen in background in parallel for
[Luke Cwik] [BEAM-12802] Refactor DataStreamsDecoder so that it becomes aware of the
[noreply] Fix typo in BigQuery documentation
[ajamato] [BEAM-11994] Instantiate a new ServiceCallMetric before each request to
[Ankur Goenka] Remove duplicate 2.33.0 section
[Steve Niemitz] [BEAM-12767] Improve PipelineOption parsing UX
[noreply] add python spark example in documentation (#15426)
[noreply] Add per-batch metrics to JdbcIO.write (#15429)
------------------------------------------
[...truncated 61.17 KB...]
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (32 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.1-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2537908 sha256=4a01f757b6960d79bda96ecd9ae8eef0600f900ccbd15f933856a448a1f2ce7d
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pycparser, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, cffi, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.17.0 azure-storage-blob-12.8.1 boto3-1.18.33 botocore-1.21.33 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.1 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.25.2 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.3 google-resumable-media-2.0.1 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0901150448.1630511435.391014/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0901150448.1630511435.391014/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0901150448.1630511435.391014/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0901150448.1630511435.391014/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0901150448.1630511435.391014/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0901150448.1630511435.391014/dataflow-****.jar in 11 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0901150448.1630511435.391014/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0901150448.1630511435.391014/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210901155035392033-4796'
createTime: '2021-09-01T15:50:48.031017Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-01_08_50_47-9146377315528361920'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0901150448'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-01T15:50:48.031017Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-01_08_50_47-9146377315528361920]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-01_08_50_47-9146377315528361920
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-01_08_50_47-9146377315528361920?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-01_08_50_47-9146377315528361920 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:47.198Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:47.198Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:57.620Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.513Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.553Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.620Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.653Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.683Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.715Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.783Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.839Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.878Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.903Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.937Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.972Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:58.996Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.025Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.065Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.096Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.130Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.161Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.188Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.210Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.239Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.276Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.307Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.353Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.379Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.434Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.459Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:50:59.492Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:51:12.529Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:51:32.798Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 1 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:51:32.818Z: JOB_MESSAGE_DETAILED: Resized **** pool to 1, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:51:43.048Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:52:10.198Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T15:52:10.232Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T16:00:02.124Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T16:00:02.167Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T16:00:02.195Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T16:00:02.218Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-01T16:00:02.241Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-01_08_50_47-9146377315528361920 after 603 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: b02abc08c2b14775a1126c042a407645 and timestamp: 1630512240.5781853:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 60
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-01_08_50_47-9146377315528361920?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 46s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/jh3aqq3dbutkw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal :
beam_PerformanceTests_PubsubIOIT_Python_Streaming #474
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/474/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #473
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/473/display/redirect?page=changes>
Changes:
[noreply] Avoid setting empty builders in proto setters
[Brian Hulette] Add back 'more' break
[Robert Bradshaw] Fix some website logos.
[chuck.yang] Run BigQuery queries with batch priority
[chuck.yang] Allow changing query priority in ReadFromBigQuery
[chuck.yang] Update CHANGES
[kawaigin] Implicitly watch and track anonymous pipeline and PCollections
[chamikaramj] Reject requests when parameter names cannot be validated unless
[chamikaramj] Updates error message
[alexander.chermenin] [BEAM-10822] Fixed typo in BigqueryClient
[zyichi] [BEAM-12898] Minor fix to Kubernetes.groovy postBuildScript.
[noreply] [BEAM-12869] Bump tensorflow from 2.5.0 to 2.5.1 in
[noreply] [BEAM-12946] Fix SmallestPerKey, add unit testing (#15567)
[zyichi] [BEAM-12908] Sickbay pubsublite.ReadWriteIT
[noreply] Relocate Go SDK breaking change note out of template into 2.33.0.
------------------------------------------
[...truncated 61.54 KB...]
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.3-py2.py3-none-any.whl (93 kB)
Collecting protobuf<4,>=3.12.2
Using cached protobuf-3.17.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.2.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.10.0-py3-none-any.whl (51 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2547583 sha256=c70c667db267d03593f5628fd0cb4c3a67944391f298d49444d6e0d25b81c8ac
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pycparser, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, cffi, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: protobuf
Found existing installation: protobuf 3.18.0
Uninstalling protobuf-3.18.0:
Successfully uninstalled protobuf-3.18.0
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.9.0 boto3-1.18.47 botocore-1.21.47 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.5 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.3 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.2.0 google-resumable-media-2.0.3 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.10.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 protobuf-3.17.3 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.25 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.7 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0924150505.1632498633.457494/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0924150505.1632498633.457494/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0924150505.1632498633.457494/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0924150505.1632498633.457494/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0924150505.1632498633.457494/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0924150505.1632498633.457494/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0924150505.1632498633.457494/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0924150505.1632498633.457494/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210924155033458445-5120'
createTime: '2021-09-24T15:50:37.958318Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-24_08_50_36-1937739938267030395'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0924150505'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-24T15:50:37.958318Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-24_08_50_36-1937739938267030395]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-24_08_50_36-1937739938267030395
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-24_08_50_36-1937739938267030395?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-24_08_50_36-1937739938267030395 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:37.029Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:37.030Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:45.197Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:45.921Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:45.950Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.029Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.066Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.099Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.131Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.201Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.261Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.308Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.354Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.380Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.414Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.461Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.505Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.545Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.581Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.621Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.654Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.689Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.715Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.753Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.798Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.897Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.932Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:46.970Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:47.016Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:47.050Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:50:47.089Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:51:11.122Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:51:27.588Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:51:27.623Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:51:58.182Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:51:58.224Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:53:21.039Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:59:55.801Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:59:55.845Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:59:55.877Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:59:55.927Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-24T15:59:55.957Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-24_08_50_36-1937739938267030395 after 604 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 9b91763b9f184120a784a91fc9e6276d and timestamp: 1632499350.5377462:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 59
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-24_08_50_36-1937739938267030395?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 15s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/ajh6p72dvl6ws
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #472
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/472/display/redirect?page=changes>
Changes:
[kileysok] Add MapState and SetState support
[Kyle Weaver] [BEAM-12898] Use new postbuildscript dsl in Flink load tests
[Kyle Weaver] run postbuild regardless of test result
[Kyle Weaver] spotless
[noreply] Merge pull request #15487 from [BEAM-12812] - Run Github Actions on GCP
[noreply] [BEAM-11097] Add SideInputCache to harness control type (#15530)
[kawaigin] Updated interactive integration test golden screenshots.
[noreply] Revert PR 15487 (BEAM-12812) (#15554)
[zyichi] [BEAM-12898] Trying out solution suggestion for JENKINS-66189 to solve
[noreply] [BEAM-12383] Adding Go SDK and Kafka IO to Gradle cross-language test
------------------------------------------
[...truncated 62.05 KB...]
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.2.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.10.0-py3-none-any.whl (51 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2547059 sha256=fc0688831297e6229b488c29a7c35fefbbeb728414ca544cd62b15ce3d8e8ac5
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
> Task :runners:google-cloud-dataflow-java:****:compileJava
> Task :sdks:python:apache_beam:testing:load_tests:installGcpTest
Installing collected packages: pyasn1, urllib3, rsa, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pycparser, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, cffi, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: protobuf
Found existing installation: protobuf 3.18.0
Uninstalling protobuf-3.18.0:
Successfully uninstalled protobuf-3.18.0
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.9.0 boto3-1.18.46 botocore-1.21.46 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.5 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.3 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.2.0 google-resumable-media-2.0.3 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.10.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 protobuf-3.17.3 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.25 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.7 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :runners:google-cloud-dataflow-java:****:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:google-cloud-dataflow-java:****:classes
> Task :runners:google-cloud-dataflow-java:****:shadowJar
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922200453.1632412326.497912/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922200453.1632412326.497912/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922200453.1632412326.497912/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922200453.1632412326.497912/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922200453.1632412326.497912/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922200453.1632412326.497912/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922200453.1632412326.497912/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922200453.1632412326.497912/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210923155206498885-6674'
createTime: '2021-09-23T15:52:11.184180Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-23_08_52_10-3003323734479457269'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0922200453'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-23T15:52:11.184180Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-23_08_52_10-3003323734479457269]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-23_08_52_10-3003323734479457269
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-23_08_52_10-3003323734479457269?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-23_08_52_10-3003323734479457269 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:10.323Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:10.323Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:17.023Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:17.651Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:17.680Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:17.748Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:17.787Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:17.833Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:17.858Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:17.932Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:17.996Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.030Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.081Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.134Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.182Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.206Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.236Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.265Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.298Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.342Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.362Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.396Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.429Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.466Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.497Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.531Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.555Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.595Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.664Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.700Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:18.737Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:44.362Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:56.672Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 2 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:52:56.711Z: JOB_MESSAGE_DETAILED: Resized **** pool to 2, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:53:07.059Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:53:34.497Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T15:53:34.529Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T16:01:21.835Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T16:01:21.887Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T16:01:21.910Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T16:01:21.949Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-23T16:01:21.985Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-23_08_52_10-3003323734479457269 after 604 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 8acad549720845eda11ca1417e5f3a2a and timestamp: 1632413038.0582361:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 62
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-23_08_52_10-3003323734479457269?project=apache-beam-testing
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 41s
83 actionable tasks: 56 executed, 27 from cache
Publishing build scan...
https://gradle.com/s/4piviyceekyfq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #471
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/471/display/redirect?page=changes>
Changes:
[noreply] [BEAM-12919] Removed IBM Streams from runner matrix (#15542)
[danthev] Fix 2.32.0 release notes.
[noreply] [BEAM-12258] Re-throw exception from forked thread in
[kawaigin] [BEAM-10708] Added an example notebook for beam_sql magic
[noreply] Add a timeout for BQ streaming_insert RPCS (#15541)
[noreply] Merge pull request #15537 from [BEAM-12908] Add a sleep to the IT after
------------------------------------------
[...truncated 61.32 KB...]
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.3-py2.py3-none-any.whl (93 kB)
Collecting protobuf<4,>=3.12.2
Using cached protobuf-3.17.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.2.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.10.0-py3-none-any.whl (51 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2547059 sha256=b92ee2ad32c2c240508e547946fa0905c7d1ee75a6d472a7c1765d32899dc907
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pycparser, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, cffi, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: protobuf
Found existing installation: protobuf 3.18.0
Uninstalling protobuf-3.18.0:
Successfully uninstalled protobuf-3.18.0
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.9.0 boto3-1.18.45 botocore-1.21.45 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.3 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.2.0 google-resumable-media-2.0.3 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.10.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 protobuf-3.17.3 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.24 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922150503.1632325833.482472/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922150503.1632325833.482472/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922150503.1632325833.482472/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922150503.1632325833.482472/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922150503.1632325833.482472/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922150503.1632325833.482472/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922150503.1632325833.482472/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0922150503.1632325833.482472/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210922155033483436-8633'
createTime: '2021-09-22T15:50:37.976064Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-22_08_50_37-11010998379505396983'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0922150503'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-22T15:50:37.976064Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-22_08_50_37-11010998379505396983]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-22_08_50_37-11010998379505396983
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-22_08_50_37-11010998379505396983?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-22_08_50_37-11010998379505396983 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:37.252Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:37.253Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:45.086Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:45.712Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:45.739Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:45.797Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:45.823Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:45.861Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:45.890Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:45.954Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.038Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.088Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.116Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.144Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.177Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.207Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.232Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.268Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.305Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.328Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.363Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.403Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.437Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.463Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.509Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.537Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.570Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.610Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.681Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.719Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:50:46.755Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:51:20.202Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:51:22.056Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:51:55.284Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:51:55.316Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:59:55.201Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:59:55.274Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:59:55.314Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:59:55.353Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-22T15:59:55.387Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-22_08_50_37-11010998379505396983 after 602 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: e9197594e4f64aaf9cc3d64b8a36b594 and timestamp: 1632326549.7433214:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 78
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-22_08_50_37-11010998379505396983?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 12s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/t6zzi4btbkoqs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #470
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/470/display/redirect?page=changes>
Changes:
[chamikaramj] Python support for directly using Java transforms using constructor and
[chamikaramj] Fixes yapf
[chamikaramj] Fixes lint
[chamikaramj] Addressing reviewer comments
[chamikaramj] Adds support for a field name format that will be ignored at expansion
[danthev] Fix flaky test.
[danthev] Fix lint errors.
[Robert Bradshaw] Additional CoGBK tests.
[chamikaramj] Addresses reviewer comments
[chamikaramj] Use correct ignore field prefix in Python side
[noreply] [BEAM-12803] Update deprecated use of _field_types (#15539)
[Robert Bradshaw] Move CoGBK tests into appropreate module.
------------------------------------------
[...truncated 61.55 KB...]
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.3-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting protobuf<4,>=3.12.2
Using cached protobuf-3.17.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.2.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.10.0-py3-none-any.whl (51 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2545631 sha256=4180b630ddb08a7f4c2fadb70b09e9afca550b5087e19e5db95126885cc64115
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pycparser, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, cffi, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: protobuf
Found existing installation: protobuf 3.18.0
Uninstalling protobuf-3.18.0:
Successfully uninstalled protobuf-3.18.0
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.9.0 boto3-1.18.44 botocore-1.21.44 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.3 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.2.0 google-resumable-media-2.0.3 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.10.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 protobuf-3.17.3 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0921150451.1632239432.611327/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0921150451.1632239432.611327/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0921150451.1632239432.611327/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0921150451.1632239432.611327/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0921150451.1632239432.611327/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0921150451.1632239432.611327/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0921150451.1632239432.611327/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0921150451.1632239432.611327/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210921155032612301-4629'
createTime: '2021-09-21T15:50:37.728062Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-21_08_50_36-5010087494035456846'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0921150451'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-21T15:50:37.728062Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-21_08_50_36-5010087494035456846]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-21_08_50_36-5010087494035456846
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-21_08_50_36-5010087494035456846?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-21_08_50_36-5010087494035456846 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:36.826Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:36.827Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:44.695Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.290Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.332Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.389Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.444Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.484Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.515Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.584Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.635Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.676Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.715Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.752Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.783Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.818Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.848Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.880Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.931Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.957Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:45.988Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.021Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.052Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.079Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.132Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.173Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.210Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.231Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.295Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.322Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:50:46.361Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:51:17.799Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:51:30.328Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:52:01.106Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:52:01.154Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:59:56.258Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:59:56.320Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:59:56.357Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:59:56.391Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-21T15:59:56.434Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-21_08_50_36-5010087494035456846 after 603 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: b65d980a4c7247dabc6459588235e8e2 and timestamp: 1632240144.3655143:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 76
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-21_08_50_36-5010087494035456846?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 10s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/vpf5d5auelltq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #469
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/469/display/redirect>
Changes:
------------------------------------------
[...truncated 61.04 KB...]
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.10.0-py3-none-any.whl (51 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2543743 sha256=b802837900d7537d5e4d76f8476019c849e2977697b81ae663ebed36f7fd4e99
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.44 botocore-1.21.44 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.10.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0920150512.1632153037.297188/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0920150512.1632153037.297188/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0920150512.1632153037.297188/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0920150512.1632153037.297188/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0920150512.1632153037.297188/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0920150512.1632153037.297188/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0920150512.1632153037.297188/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0920150512.1632153037.297188/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210920155037298197-3640'
createTime: '2021-09-20T15:50:41.789427Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-20_08_50_40-6451490653412522242'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0920150512'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-20T15:50:41.789427Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-20_08_50_40-6451490653412522242]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-20_08_50_40-6451490653412522242
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-20_08_50_40-6451490653412522242?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-20_08_50_40-6451490653412522242 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:40.890Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:40.890Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:48.996Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:49.796Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:49.829Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:49.897Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:49.933Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:49.969Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.060Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.117Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.158Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.187Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.221Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.249Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.281Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.315Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.343Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.378Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.402Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.438Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.471Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.508Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.541Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.574Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.611Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.665Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.691Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.742Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.775Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:50.797Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:50:59.795Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:51:34.073Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:52:03.775Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T15:52:03.806Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T16:00:03.798Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T16:00:03.845Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T16:00:03.865Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T16:00:03.894Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-20T16:00:03.917Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-20_08_50_40-6451490653412522242 after 604 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 1e966fab0d6546ee9cd900663e14ea09 and timestamp: 1632153758.769092:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 76
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-20_08_50_40-6451490653412522242?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 21s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/wqbyvrmiyigyu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #468
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/468/display/redirect>
Changes:
------------------------------------------
[...truncated 60.84 KB...]
Using cached google_cloud_bigtable-1.5.0-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.10.0-py3-none-any.whl (51 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2543743 sha256=9a32feed900903b404b9234108ceb5d0976c2e28be8cec7b78877249960e1887
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.44 botocore-1.21.44 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.10.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0919150455.1632066621.193982/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0919150455.1632066621.193982/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0919150455.1632066621.193982/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0919150455.1632066621.193982/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0919150455.1632066621.193982/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0919150455.1632066621.193982/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0919150455.1632066621.193982/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0919150455.1632066621.193982/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210919155021194985-3226'
createTime: '2021-09-19T15:50:26.103542Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-19_08_50_24-4554860190093149615'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0919150455'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-19T15:50:26.103542Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-19_08_50_24-4554860190093149615]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-19_08_50_24-4554860190093149615
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-19_08_50_24-4554860190093149615?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-19_08_50_24-4554860190093149615 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:25.021Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:25.021Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:33.664Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.548Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.583Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.658Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.691Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.731Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.771Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.834Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.887Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.930Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.959Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:34.991Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.024Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.061Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.094Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.128Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.156Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.183Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.218Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.254Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.288Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.313Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.343Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.376Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.410Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.443Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.526Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.557Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:50:35.587Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:51:05.539Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:51:19.091Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:51:50.009Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:51:50.037Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:59:43.988Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:59:44.084Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:59:44.104Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:59:44.140Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-19T15:59:44.174Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-19_08_50_24-4554860190093149615 after 604 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 34603097aca14edb9f4a33ffd7b0c977 and timestamp: 1632067338.1549675:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 70
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-19_08_50_24-4554860190093149615?project=apache-beam-testing
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 2s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/gjnkwgtppf4cs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #467
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/467/display/redirect?page=changes>
Changes:
[samuelw] [BEAM-12740] Add option to CreateOptions to avoid GetObjectMetadata for
[noreply] Minor: Prune docker volumes in Inventory job(#15532)
------------------------------------------
[...truncated 61.47 KB...]
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.10.0-py3-none-any.whl (51 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2543743 sha256=9d84f0c740b866f2f3e6f01f31cdb765a58358d7ea169c76c1f527b3ab8ed828
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.44 botocore-1.21.44 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.10.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0918150447.1631980256.062278/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0918150447.1631980256.062278/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0918150447.1631980256.062278/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0918150447.1631980256.062278/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0918150447.1631980256.062278/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0918150447.1631980256.062278/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0918150447.1631980256.062278/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0918150447.1631980256.062278/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210918155056063370-7878'
createTime: '2021-09-18T15:51:01.231224Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-18_08_51_00-7949284943934414126'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0918150447'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-18T15:51:01.231224Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-18_08_51_00-7949284943934414126]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-18_08_51_00-7949284943934414126
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-18_08_51_00-7949284943934414126?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-18_08_51_00-7949284943934414126 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:00.297Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:00.298Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:08.354Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:08.981Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.006Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.070Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.092Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.127Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.155Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.223Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.280Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.316Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.344Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.378Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.409Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.438Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.469Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.497Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.524Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.556Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.578Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.621Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.644Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.667Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.695Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.720Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.743Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.770Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.823Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.854Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:09.896Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:32.515Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:40.424Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 1 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:40.446Z: JOB_MESSAGE_DETAILED: Resized **** pool to 1, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:51:50.719Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:52:20.350Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T15:52:20.374Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T16:00:16.161Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T16:00:16.201Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T16:00:16.242Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T16:00:16.275Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-18T16:00:16.307Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-18_08_51_00-7949284943934414126 after 602 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: c420246a6da74ac782988c0857388167 and timestamp: 1631980967.558237:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 80
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-18_08_51_00-7949284943934414126?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 28s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/wafmhltg7faoe
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #466
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/466/display/redirect?page=changes>
Changes:
[randomstep] [BEAM-12899] Upgrade Gradle to version 6.9.x
[noreply] [BEAM-12701] Added extra parameter in to_csv for DeferredFrame to name
[zyichi] [BEAM-12603] Add retries to FnApiRunnerTest due to flakiness of grpc
[noreply] [BEAM-12535] add dataframes notebook (#15470)
------------------------------------------
[...truncated 60.89 KB...]
Using cached google_cloud_bigtable-1.6.0-py2.py3-none-any.whl (267 kB)
Using cached google_cloud_bigtable-1.5.1-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.5.0-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.10.0-py3-none-any.whl (51 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.5-py3-none-any.whl (37 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2543743 sha256=2dba6a4e698bc085db7aa28610c542f3d56065c3d305f1c16255989a870fff53
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.43 botocore-1.21.43 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.5 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.10.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0917150507.1631893832.032486/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0917150507.1631893832.032486/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0917150507.1631893832.032486/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0917150507.1631893832.032486/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0917150507.1631893832.032486/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0917150507.1631893832.032486/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0917150507.1631893832.032486/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0917150507.1631893832.032486/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210917155032033524-4669'
createTime: '2021-09-17T15:50:36.733380Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-17_08_50_35-9001999777665819081'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0917150507'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-17T15:50:36.733380Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-17_08_50_35-9001999777665819081]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-17_08_50_35-9001999777665819081
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-17_08_50_35-9001999777665819081?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-17_08_50_35-9001999777665819081 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:35.935Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:35.935Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:43.432Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.200Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.231Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.289Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.312Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.340Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.374Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.422Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.491Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.534Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.562Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.593Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.631Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.665Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.700Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.731Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.764Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.789Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.816Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.842Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.868Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.902Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.946Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:44.981Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:45.018Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:45.053Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:45.110Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:45.141Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:45.190Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:50:55.650Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:51:25.110Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:51:55.004Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T15:51:55.037Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T16:00:02.513Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T16:00:02.567Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T16:00:02.598Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T16:00:02.635Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-17T16:00:02.669Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-17_08_50_35-9001999777665819081 after 603 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 714edf7299aa44f1adfd5d9496a89d57 and timestamp: 1631894555.8228872:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 79
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-17_08_50_35-9001999777665819081?project=apache-beam-testing
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 21s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/io46u272ylat6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #465
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/465/display/redirect?page=changes>
Changes:
[dpcollins] [BEAM-12882] - fix test that is flaky when jenkins is overloaded
[noreply] [BEAM-12885] Enable NeedsRunner Tests for Samza Portable Runner (#15512)
[noreply] [BEAM-12100][BEAM-10379][BEAM-9514][BEAM-12647][BEAM-12099]
[noreply] [BEAM-12543] Fix DataFrrame typo (#15509)
[noreply] [BEAM-12794] Remove obsolete uses of sys.exc_info. (#15507)
[noreply] [BEAM-11666] flake on RecordingManagerTest (#15118)
[kawaigin] [BEAM-10708] Introspect beam_sql output
[noreply] Minor: Restore "Bugfix" section in CHANGES.md (#15516)
[Kyle Weaver] [BEAM-10459] Unignore numeric aggregation tests.
[ajamato] [BEAM-12898] Disable Flink Load tests which are leading Dataproc
------------------------------------------
[...truncated 61.25 KB...]
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.5-py3-none-any.whl (37 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2543448 sha256=d999e533edfe77e19242a752fa7524a7cf54b7fc4c49e3dc58bb8e6f2e57ec82
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.42 botocore-1.21.42 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.5 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0916150510.1631807436.296398/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0916150510.1631807436.296398/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0916150510.1631807436.296398/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0916150510.1631807436.296398/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0916150510.1631807436.296398/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0916150510.1631807436.296398/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0916150510.1631807436.296398/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0916150510.1631807436.296398/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210916155036297854-2571'
createTime: '2021-09-16T15:50:41.351413Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-16_08_50_40-3140186426580453321'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0916150510'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-16T15:50:41.351413Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-16_08_50_40-3140186426580453321]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-16_08_50_40-3140186426580453321
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-16_08_50_40-3140186426580453321?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-16_08_50_40-3140186426580453321 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:50:40.258Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:50:40.259Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:50:56.842Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:50:58.142Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:50:58.285Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:50:58.684Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:50:58.963Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:50:59.280Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:50:59.551Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:00.078Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:00.538Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:00.686Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:00.798Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:00.831Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:00.865Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:00.904Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:00.934Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:00.971Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.006Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.042Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.078Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.106Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.178Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.222Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.273Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.313Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.380Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:01.648Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:02.195Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:02.229Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:02.278Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:27.001Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:33.867Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:33.901Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:51:44.154Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:52:10.418Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T15:52:10.455Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T16:00:13.860Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T16:00:13.920Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T16:00:13.949Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T16:00:13.981Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-16T16:00:14.024Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-16_08_50_40-3140186426580453321 after 600 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: b8ef632e35cc40798433289678d7e734 and timestamp: 1631808168.7608352:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 98
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-16_08_50_40-3140186426580453321?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 646, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 656, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 702, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 31s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/cvlj6as4mhtpy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #464
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/464/display/redirect?page=changes>
Changes:
[dhuntsperger] updated Maven-to-Gradle conversion step in Java quickstart
[noreply] [BEAM-10913] - Updating Grafana from v6.7.3 to v8.1.2 (#15503)
[noreply] [BEAM-12876] Adding doc and glossary entry for resource hints (#15499)
[noreply] [BEAM-12153] revert "implement GroupByKey with CombinePerKey with
[noreply] [BEAM-12845] Add AWS services as a runtime dependency to Spark Job
------------------------------------------
[...truncated 61.75 KB...]
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.5-py3-none-any.whl (37 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2540921 sha256=a017deaceeefcde59ddfed3c36c437623a80aba2fd7bda9a4c6eaeea34bb0d38
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.42 botocore-1.21.42 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.5 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :runners:google-cloud-dataflow-java:****:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:google-cloud-dataflow-java:****:classes
> Task :runners:google-cloud-dataflow-java:****:shadowJar
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0915150504.1631721177.060412/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0915150504.1631721177.060412/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0915150504.1631721177.060412/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0915150504.1631721177.060412/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0915150504.1631721177.060412/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0915150504.1631721177.060412/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0915150504.1631721177.060412/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0915150504.1631721177.060412/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210915155257061436-2358'
createTime: '2021-09-15T15:53:01.840680Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-15_08_53_01-820026579061817356'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0915150504'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-15T15:53:01.840680Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-15_08_53_01-820026579061817356]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-15_08_53_01-820026579061817356
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-15_08_53_01-820026579061817356?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-15_08_53_01-820026579061817356 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:01.063Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:01.064Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:07.580Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.230Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.264Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.327Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.361Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.421Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.448Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.505Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.571Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.630Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.658Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.685Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.709Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.747Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.783Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.818Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.853Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.886Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.922Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.947Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:08.981Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:09.016Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:09.061Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:09.098Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:09.130Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:09.166Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:09.215Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:09.235Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:09.274Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:43.164Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:53:53.105Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:54:22.503Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T15:54:22.541Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T16:02:22.410Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T16:02:22.483Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T16:02:22.510Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T16:02:22.549Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-15T16:02:22.585Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-15_08_53_01-820026579061817356 after 604 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 5be1c420210443bba0f1ef82d77bbf10 and timestamp: 1631721898.6266122:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 54
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-15_08_53_01-820026579061817356?project=apache-beam-testing
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 643, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 653, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 699, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 643, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 653, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 699, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 699, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 15m 39s
83 actionable tasks: 55 executed, 28 from cache
Publishing build scan...
https://gradle.com/s/rq7a4wtttiap4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #463
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/463/display/redirect?page=changes>
Changes:
[rohde.samuel] [BEAM-12842] Add timestamp to test work item to deflake
[dhuntsperger] fixed broken Python tab on HCatalog IO page
[suztomo] [BEAM-12873] HL7v2IO: to leave schematizedData null, not empty
[noreply] Avoid apiary submission of job graph when it is not needed. (#15458)
[noreply] [BEAM-7261] Add support for BasicSessionCredentials for AWS credentials.
[noreply] Bump dataflow java container version to beam-master-20210913 (#15506)
[noreply] [BEAM-11980] Java GCS - Implement IO Request Count metrics (#15394)
------------------------------------------
[...truncated 60.99 KB...]
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2540921 sha256=0de125f6bc2b5d224d087694f0064dd36b35900e53619772bb3e6ea425e2a6eb
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.41 botocore-1.21.41 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0914150520.1631634623.075733/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0914150520.1631634623.075733/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0914150520.1631634623.075733/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0914150520.1631634623.075733/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0914150520.1631634623.075733/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0914150520.1631634623.075733/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0914150520.1631634623.075733/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0914150520.1631634623.075733/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210914155023076849-9953'
createTime: '2021-09-14T15:50:28.182912Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-14_08_50_27-14182980976911616920'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0914150520'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-14T15:50:28.182912Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-14_08_50_27-14182980976911616920]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-14_08_50_27-14182980976911616920
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-14_08_50_27-14182980976911616920?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-14_08_50_27-14182980976911616920 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:27.234Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:27.234Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:34.793Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:35.776Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:35.812Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:35.871Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:35.900Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:35.924Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:35.951Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.034Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.091Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.133Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.170Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.200Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.235Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.263Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.296Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.331Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.359Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.397Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.428Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.459Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.497Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.528Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.569Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.597Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.632Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.663Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.711Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.753Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:36.787Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:50:49.763Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:51:18.704Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:51:48.090Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:51:48.115Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:59:47.346Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:59:47.377Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:59:47.409Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:59:47.453Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-14T15:59:47.478Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-14_08_50_27-14182980976911616920 after 602 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 75a120291df44074bc4acdf03f01631a and timestamp: 1631635350.072041:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 78
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-14_08_50_27-14182980976911616920?project=apache-beam-testing
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 643, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 653, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 699, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 643, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 653, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 699, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 699, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 833, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 13s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/f73kup2vvwcpy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #462
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/462/display/redirect>
Changes:
------------------------------------------
[...truncated 55.69 KB...]
Using cached google_cloud_pubsub-1.7.0-py2.py3-none-any.whl (144 kB)
Collecting google-cloud-bigquery<3,>=1.6.0
Using cached google_cloud_bigquery-2.26.0-py2.py3-none-any.whl (201 kB)
Collecting google-cloud-core<2,>=0.28.1
Using cached google_cloud_core-1.7.2-py2.py3-none-any.whl (28 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.7.0-py2.py3-none-any.whl (267 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.19.1-py2.py3-none-any.whl (255 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.16.1-py2.py3-none-any.whl (183 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-recommendations-ai<=0.2.0,>=0.1.0
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting mock<3.0.0,>=1.0.1
Using cached mock-2.0.0-py2.py3-none-any.whl (56 kB)
Collecting pandas<1.3.0,>=1.0
Using cached pandas-1.2.5-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (9.9 MB)
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.5-py2.py3-none-any.whl (17 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Collecting pyyaml<6.0.0,>=3.12
Using cached PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.9.3-py2.py3-none-any.whl (27 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.4.23-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)
Collecting testcontainers<4.0.0,>=3.0.3
Using cached testcontainers-3.4.2-py2.py3-none-any.whl (31 kB)
Collecting boto3>=1.9
Using cached boto3-1.18.40-py3-none-any.whl (131 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.34.0.dev0) (1.16.0)
Collecting cryptography>=2.1.4
Using cached cryptography-3.4.8-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
Collecting msrest>=0.6.18
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting botocore<1.22.0,>=1.21.40
Using cached botocore-1.21.40-py3-none-any.whl (7.9 MB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.0-py3-none-any.whl (79 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting urllib3<1.27,>=1.25.4
Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting cffi>=1.12
Using cached cffi-1.14.6-cp37-cp37m-manylinux1_x86_64.whl (402 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.16.3-py2.py3-none-any.whl (28 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.34.0.dev0) (58.0.4)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.34.0.dev0) (21.0)
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.1-py2.py3-none-any.whl (92 kB)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.19.0-py3-none-any.whl (42 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.0.2-py2.py3-none-any.whl (75 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.6.1-py2.py3-none-any.whl (267 kB)
Using cached google_cloud_bigtable-1.6.0-py2.py3-none-any.whl (267 kB)
Using cached google_cloud_bigtable-1.5.1-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.5.0-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2540821 sha256=ed9102cb2e4839ad7e08bb5f7d758d39af3d4cff9361b013acfe963235e697fc
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.40 botocore-1.21.40 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0913150523.1631548235.039537/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0913150523.1631548235.039537/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0913150523.1631548235.039537/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0913150523.1631548235.039537/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0913150523.1631548235.039537/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0913150523.1631548235.039537/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0913150523.1631548235.039537/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0913150523.1631548235.039537/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210913155035040551-9999'
createTime: '2021-09-13T15:50:39.815214Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-13_08_50_38-17875373317011179196'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0913150523'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-13T15:50:39.815214Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-13_08_50_38-17875373317011179196]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-13_08_50_38-17875373317011179196
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-13_08_50_38-17875373317011179196?project=apache-beam-testing
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-13_08_50_38-17875373317011179196?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 222, in <module>
PubsubWritePerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 149, in run
self.result = self.pipeline.run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/test_pipeline.py",> line 118, in run
"Pipeline execution failed."
AssertionError: Pipeline execution failed.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1m 37s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/nw3wefm4zltdq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #461
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/461/display/redirect>
Changes:
------------------------------------------
[...truncated 56.09 KB...]
Collecting google-cloud-core<2,>=0.28.1
Using cached google_cloud_core-1.7.2-py2.py3-none-any.whl (28 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.7.0-py2.py3-none-any.whl (267 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.19.1-py2.py3-none-any.whl (255 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.16.1-py2.py3-none-any.whl (183 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-recommendations-ai<=0.2.0,>=0.1.0
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Collecting boto3>=1.9
Using cached boto3-1.18.40-py3-none-any.whl (131 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting mock<3.0.0,>=1.0.1
Using cached mock-2.0.0-py2.py3-none-any.whl (56 kB)
Collecting pandas<1.3.0,>=1.0
Using cached pandas-1.2.5-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (9.9 MB)
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.5-py2.py3-none-any.whl (17 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Collecting pyyaml<6.0.0,>=3.12
Using cached PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.9.3-py2.py3-none-any.whl (27 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.4.23-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)
Collecting testcontainers<4.0.0,>=3.0.3
Using cached testcontainers-3.4.2-py2.py3-none-any.whl (31 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.34.0.dev0) (1.16.0)
Collecting cryptography>=2.1.4
Using cached cryptography-3.4.8-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
Collecting msrest>=0.6.18
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.0-py3-none-any.whl (79 kB)
Collecting botocore<1.22.0,>=1.21.40
Using cached botocore-1.21.40-py3-none-any.whl (7.9 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting urllib3<1.27,>=1.25.4
Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting cffi>=1.12
Using cached cffi-1.14.6-cp37-cp37m-manylinux1_x86_64.whl (402 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.16.3-py2.py3-none-any.whl (28 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.34.0.dev0) (58.0.4)
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.1-py2.py3-none-any.whl (92 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.0.2-py2.py3-none-any.whl (75 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.34.0.dev0) (21.0)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.19.0-py3-none-any.whl (42 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.6.1-py2.py3-none-any.whl (267 kB)
Using cached google_cloud_bigtable-1.6.0-py2.py3-none-any.whl (267 kB)
Using cached google_cloud_bigtable-1.5.1-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.5.0-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2540821 sha256=7a0fa76cf66b27034240f9c3a78f4f3ec598c7874fc90f22bf2a82c353742105
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.40 botocore-1.21.40 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0912150444.1631461840.955445/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0912150444.1631461840.955445/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0912150444.1631461840.955445/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0912150444.1631461840.955445/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0912150444.1631461840.955445/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0912150444.1631461840.955445/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0912150444.1631461840.955445/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0912150444.1631461840.955445/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210912155040956499-6372'
createTime: '2021-09-12T15:50:45.711058Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-12_08_50_44-8645107419101515445'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0912150444'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-12T15:50:45.711058Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-12_08_50_44-8645107419101515445]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-12_08_50_44-8645107419101515445
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-12_08_50_44-8645107419101515445?project=apache-beam-testing
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 222, in <module>
PubsubWritePerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 149, in run
self.result = self.pipeline.run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/test_pipeline.py",> line 118, in run
"Pipeline execution failed."
AssertionError: Pipeline execution failed.
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-12_08_50_44-8645107419101515445?project=apache-beam-testing
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1m 42s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/eynyqtrgkyjyo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #460
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/460/display/redirect?page=changes>
Changes:
[noreply] Added type annotations to some combiners missing it. (#15414)
[noreply] [BEAM-12634] JmsIO auto scaling feature (#15464)
[noreply] [BEAM-12662] Get Flink version from cluster. (#15223)
[noreply] Port changes from Pub/Sub Lite to beam (#15418)
[heejong] [BEAM-12805] Fix XLang CombinePerKey test by explicitly assigning the
[BenWhitehead] [BEAM-8376] Google Cloud Firestore Connector - Add handling for
[noreply] Decreasing peak memory usage for beam.TupleCombineFn (#15494)
[noreply] [BEAM-12802] Add support for prefetch through data layers down through
[noreply] [BEAM-11097] Add implementation of side input cache (#15483)
------------------------------------------
[...truncated 55.84 KB...]
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.9.3-py2.py3-none-any.whl (27 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.4.23-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)
Collecting testcontainers<4.0.0,>=3.0.3
Using cached testcontainers-3.4.2-py2.py3-none-any.whl (31 kB)
Collecting boto3>=1.9
Using cached boto3-1.18.40-py3-none-any.whl (131 kB)
Collecting cachetools<5,>=3.1.0
Using cached cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting google-apitools<0.5.32,>=0.5.31
Using cached google_apitools-0.5.31-py3-none-any.whl
Collecting google-auth<3,>=1.18.0
Using cached google_auth-2.0.2-py2.py3-none-any.whl (152 kB)
Collecting google-cloud-datastore<2,>=1.8.0
Using cached google_cloud_datastore-1.15.3-py2.py3-none-any.whl (134 kB)
Collecting google-cloud-pubsub<2,>=0.39.0
Using cached google_cloud_pubsub-1.7.0-py2.py3-none-any.whl (144 kB)
Collecting google-cloud-bigquery<3,>=1.6.0
Using cached google_cloud_bigquery-2.26.0-py2.py3-none-any.whl (201 kB)
Collecting google-cloud-core<2,>=0.28.1
Using cached google_cloud_core-1.7.2-py2.py3-none-any.whl (28 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.7.0-py2.py3-none-any.whl (267 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.19.1-py2.py3-none-any.whl (255 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.16.1-py2.py3-none-any.whl (183 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-recommendations-ai<=0.2.0,>=0.1.0
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.34.0.dev0) (1.16.0)
Collecting cryptography>=2.1.4
Using cached cryptography-3.4.8-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
Collecting msrest>=0.6.18
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.0-py3-none-any.whl (79 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.22.0,>=1.21.40
Using cached botocore-1.21.40-py3-none-any.whl (7.9 MB)
Collecting urllib3<1.27,>=1.25.4
Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting cffi>=1.12
Using cached cffi-1.14.6-cp37-cp37m-manylinux1_x86_64.whl (402 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.16.3-py2.py3-none-any.whl (28 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.34.0.dev0) (58.0.4)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting proto-plus>=1.10.0
Using cached proto_plus-1.19.0-py3-none-any.whl (42 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.0.2-py2.py3-none-any.whl (75 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.34.0.dev0) (21.0)
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.1-py2.py3-none-any.whl (92 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.6.1-py2.py3-none-any.whl (267 kB)
Using cached google_cloud_bigtable-1.6.0-py2.py3-none-any.whl (267 kB)
Using cached google_cloud_bigtable-1.5.1-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.5.0-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2540821 sha256=22f61001793264cfeafa86099ac5e705b266bab8587de592a38a98b5670e6e8f
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.40 botocore-1.21.40 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0911150515.1631375443.420103/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0911150515.1631375443.420103/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0911150515.1631375443.420103/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0911150515.1631375443.420103/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0911150515.1631375443.420103/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0911150515.1631375443.420103/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0911150515.1631375443.420103/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0911150515.1631375443.420103/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210911155043421245-8243'
createTime: '2021-09-11T15:50:48.409262Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-11_08_50_47-5284744639139526434'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0911150515'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-11T15:50:48.409262Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-11_08_50_47-5284744639139526434]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-11_08_50_47-5284744639139526434
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-11_08_50_47-5284744639139526434?project=apache-beam-testing
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 222, in <module>
PubsubWritePerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 149, in run
self.result = self.pipeline.run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/test_pipeline.py",> line 118, in run
"Pipeline execution failed."
AssertionError: Pipeline execution failed.
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-11_08_50_47-5284744639139526434?project=apache-beam-testing
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1m 44s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/6366wpt52pujw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #459
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/459/display/redirect?page=changes>
Changes:
[ruwan.lambrichts] Clarify additional_bq_parameters argument
[kawaigin] [BEAM-10708] Support streaming cache in beam_sql magic
[noreply] Fix broken 'differences from pandas' link
[noreply] Added GroupBy row in Aggregation table.
[Luke Cwik] [BEAM-12769] Fix typo in test class name, CLass -> Class
[Etienne Chauchot] [BEAM-5172] Temporary ignore testSplit and testSizes tests waiting for a
[samuelw] [BEAM-12740] Remove matching to filter files when renaming gcs files in
[noreply] [BEAM-3304] Helper functions for triggers (#15430)
[esert] Bump a throttling counter on BigQueryRead retries due to
[noreply] [BEAM-5097] Increment counter for "small words" in go SDK example
[noreply] Register MapCoder, some comments/cleanup. (#15471)
[noreply] [BEAM-12588] Multimap user state proto changes (#15473)
------------------------------------------
[...truncated 61.20 KB...]
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
> Task :runners:google-cloud-dataflow-java:****:compileJava
> Task :sdks:python:apache_beam:testing:load_tests:installGcpTest
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2539770 sha256=91ccd94431a7d811fb2e4ff92b01cd5251bb81505a11ea7a643551156213a0fe
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.39 botocore-1.21.39 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :runners:google-cloud-dataflow-java:****:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:google-cloud-dataflow-java:****:classes
> Task :runners:google-cloud-dataflow-java:****:shadowJar
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0910150520.1631289143.806998/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0910150520.1631289143.806998/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0910150520.1631289143.806998/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0910150520.1631289143.806998/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0910150520.1631289143.806998/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0910150520.1631289143.806998/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0910150520.1631289143.806998/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0910150520.1631289143.806998/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210910155223808130-1519'
createTime: '2021-09-10T15:52:28.729859Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-10_08_52_27-1190516439209682833'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0910150520'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-10T15:52:28.729859Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-10_08_52_27-1190516439209682833]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-10_08_52_27-1190516439209682833
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-10_08_52_27-1190516439209682833?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-10_08_52_27-1190516439209682833 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:27.801Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:27.801Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:37.298Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.098Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.134Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.192Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.236Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.277Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.318Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.392Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.450Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.489Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.517Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.552Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.589Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.640Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.678Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.704Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.726Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.755Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.784Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.850Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.883Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:38.929Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:39.010Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:39.051Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:39.097Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:39.126Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:39.187Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:39.216Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:52:39.251Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:53:07.729Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:53:18.457Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:53:48.155Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T15:53:48.187Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T16:01:45.970Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T16:01:46.009Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T16:01:46.039Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T16:01:46.095Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-10T16:01:46.129Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-10_08_52_27-1190516439209682833 after 605 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: ad5c091bb243419a92b22c81525191a9 and timestamp: 1631289850.91072:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 70
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-10_08_52_27-1190516439209682833?project=apache-beam-testing
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 643, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 653, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 699, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 643, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 653, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 699, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 699, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 54s
83 actionable tasks: 55 executed, 28 from cache
Publishing build scan...
https://gradle.com/s/xoxbkixkxufhk
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #458
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/458/display/redirect?page=changes>
Changes:
[suztomo] [BEAM-11205] Upgrading the Libraries BOM to v22
[randomstep] [BEAM-12708] Bump arrow-memory-netty
[Etienne Chauchot] [BEAM-12153] implement GroupByKey with CombinePerKey with Concatenate
[Etienne Chauchot] [BEAM-11023] Increase memory in SS Validates runner tests to avoid OOM
[Etienne Chauchot] [BEAM-12727] extract Concatenate CombineFn to runner-core module to
[noreply] Add display data for JdbcIO.write (#15460)
[noreply] Merge pull request #15480: [BEAM-12356] Make sure DatasetService is
[noreply] [BEAM-11981] Java Bigtable - Implement IO Request Count metrics (#15342)
[noreply] [BEAM-12834] Improve Go SDK cross-language documentation and API.
------------------------------------------
[...truncated 62.03 KB...]
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
> Task :runners:google-cloud-dataflow-java:****:compileJava
> Task :sdks:python:apache_beam:testing:load_tests:installGcpTest
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2539331 sha256=0a5b123d157fe61dedeaaa026b3cb62e2ecd912a4454edd4dfd1584cfe437139
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.38 botocore-1.21.38 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :runners:google-cloud-dataflow-java:****:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:google-cloud-dataflow-java:****:classes
> Task :runners:google-cloud-dataflow-java:****:shadowJar
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0909150452.1631202735.140783/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0909150452.1631202735.140783/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0909150452.1631202735.140783/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0909150452.1631202735.140783/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0909150452.1631202735.140783/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0909150452.1631202735.140783/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0909150452.1631202735.140783/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0909150452.1631202735.140783/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210909155215141944-3519'
createTime: '2021-09-09T15:52:19.965362Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-09_08_52_19-10471083336861777763'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0909150452'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-09T15:52:19.965362Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-09_08_52_19-10471083336861777763]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-09_08_52_19-10471083336861777763
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-09_08_52_19-10471083336861777763?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-09_08_52_19-10471083336861777763 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:19.195Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:19.195Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:27.647Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.463Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.493Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.554Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.592Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.627Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.646Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.690Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.740Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.782Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.813Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.848Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.874Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.893Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.920Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:28.992Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.019Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.054Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.086Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.111Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.131Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.153Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.192Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.273Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.306Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.353Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.411Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.443Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:29.475Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:52:55.437Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:53:15.779Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:53:46.075Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T15:53:46.094Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T16:00:01.987Z: JOB_MESSAGE_ERROR: There were errors attempting to publish to topic projects/apache-beam-testing/topics/pubsub_io_performance_8c3cefd3-462c-49b8-b963-ff8c89f0d5be. Recent publish calls completed successfully: 422, recent publish calls completed with error: 2.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T16:01:35.751Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T16:01:35.815Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T16:01:35.887Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T16:01:35.925Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-09T16:01:35.978Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-09_08_52_19-10471083336861777763 after 600 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 126379303d1c449b9416ad7eb5fc06b9 and timestamp: 1631203447.1299374:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 60
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-09_08_52_19-10471083336861777763?project=apache-beam-testing
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 51s
83 actionable tasks: 55 executed, 28 from cache
Publishing build scan...
Publishing failed.
The build scan server appears to be unavailable.
Please check https://status.gradle.com for the latest service status.
If the service is reported as available, please report this problem via https://gradle.com/help/plugin and include the following via copy/paste:
----------
Gradle version: 6.8.3
Plugin version: 3.4.1
Request URL: https://status.gradle.com
Request ID: 2d7c12f4-5f6c-4a79-9fdf-59bfd194eca8
Response status code: 405
Response server type: Varnish
----------
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #457
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/457/display/redirect?page=changes>
Changes:
[heejong] [BEAM-12838] Update artifact local path for DataflowRunner Java
[aydar.zaynutdinov] [BEAM-3385] Add requires about `equals()` and `hashMethod()` to
[aydar.zaynutdinov] [BEAM-3385] Changes regarding spotlessApply task
[Valentyn Tymofieiev] Disable Kafka perf tests.
[heejong] separate into resolveArtifacts method
[heejong] add test
[aydar.zaynutdinov] [BEAM-3385] wrap up equals() and hashCode() methods into links
[Andrew Pilloud] [BEAM-12850] Calcite drops empty Calc now
[Andrew Pilloud] [BEAM-12853] VALUES produces a UNION, window can't be set afterwards
[Andrew Pilloud] [BEAM-12852] Revert BigTable changes, just cast to bigint
[heejong] update
[heejong] fix formatting
[Andrew Pilloud] [BEAM-12851] Map output table names
[Luke Cwik] [BEAM-12802] Define a prefetchable iterator and iterable and utility
[vincent.marquez] [BEAM-9008] adds CassandraIO.readAll
------------------------------------------
[...truncated 61.21 KB...]
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2539186 sha256=70cd6c9dc668a9f3518169bd0f95674216cfb4184af53b9f8d358919cbc092ae
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.37 botocore-1.21.37 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0908150533.1631116229.488783/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0908150533.1631116229.488783/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0908150533.1631116229.488783/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0908150533.1631116229.488783/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0908150533.1631116229.488783/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0908150533.1631116229.488783/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0908150533.1631116229.488783/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0908150533.1631116229.488783/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210908155029489882-2215'
createTime: '2021-09-08T15:50:34.427661Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-08_08_50_33-2339281448012400280'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0908150533'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-08T15:50:34.427661Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-08_08_50_33-2339281448012400280]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-08_08_50_33-2339281448012400280
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-08_08_50_33-2339281448012400280?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-08_08_50_33-2339281448012400280 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:33.605Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:33.606Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:41.810Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.415Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.444Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.503Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.556Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.600Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.635Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.700Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.754Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.792Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.837Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.872Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.901Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.924Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.949Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:42.972Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.007Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.035Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.066Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.110Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.152Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.186Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.220Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.256Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.289Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.312Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.381Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.426Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:50:43.460Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:51:16.861Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:51:20.334Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 1 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:51:20.364Z: JOB_MESSAGE_DETAILED: Resized **** pool to 1, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:51:30.697Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:51:57.519Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:51:57.555Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:59:51.587Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:59:51.643Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:59:51.672Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:59:51.697Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-08T15:59:51.733Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-08_08_50_33-2339281448012400280 after 600 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 8bbab78d15dc4658bd9457b616444436 and timestamp: 1631116936.1622343:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 104
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-08_08_50_33-2339281448012400280?project=apache-beam-testing
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 12m 58s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/rkgisjt6wcv5q
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #456
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/456/display/redirect?page=changes>
Changes:
[kwu] [BEAM-12828] Convert UseTestStream tests to use Long instead of Integer
[kwu] Apply SpotlessJava
[kwu] Apply SpotlessJava
[noreply] Update runners/flink/job-server/flink_job_server.gradle
------------------------------------------
[...truncated 61.10 KB...]
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2539186 sha256=dd635fd1299b28ee369741e7b54126f1a58d45902a85ad4547a32a2adadce176
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.36 botocore-1.21.36 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0907150512.1631029848.679957/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0907150512.1631029848.679957/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0907150512.1631029848.679957/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0907150512.1631029848.679957/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0907150512.1631029848.679957/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0907150512.1631029848.679957/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0907150512.1631029848.679957/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0907150512.1631029848.679957/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210907155048681146-8937'
createTime: '2021-09-07T15:50:53.976494Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-07_08_50_52-12010823091226061566'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0907150512'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-07T15:50:53.976494Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-07_08_50_52-12010823091226061566]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-07_08_50_52-12010823091226061566
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-07_08_50_52-12010823091226061566?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-07_08_50_52-12010823091226061566 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:50:53.047Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:50:53.047Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:01.212Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.144Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.201Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.306Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.337Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.374Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.397Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.506Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.679Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.880Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.922Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:02.971Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.014Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.047Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.078Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.135Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.169Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.197Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.229Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.251Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.287Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.319Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.352Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.380Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.421Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.458Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.500Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.530Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:03.561Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:20.871Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:51:43.966Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:52:12.787Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T15:52:12.813Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T16:00:09.676Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T16:00:09.711Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T16:00:09.743Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T16:00:09.778Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-07T16:00:09.810Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-07_08_50_52-12010823091226061566 after 603 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: f8e2c492a46043f0961384d8a52b0525 and timestamp: 1631030560.8416345:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 107
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-07_08_50_52-12010823091226061566?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 19s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/2vulwe4g6f5pg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #455
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/455/display/redirect?page=changes>
Changes:
[noreply] Merge pull request #14927 from [BEAM-12400] MongoDBIO support for update
------------------------------------------
[...truncated 61.29 KB...]
Using cached google_cloud_bigtable-1.5.0-py2.py3-none-any.whl (266 kB)
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Using cached google_cloud_bigtable-1.3.0-py2.py3-none-any.whl (259 kB)
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2539186 sha256=d7aff11cd5b393cf1443313a800fc597627e46a72f43e084120e2c5b76e50a2d
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.36 botocore-1.21.36 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0906150541.1630943439.623985/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0906150541.1630943439.623985/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0906150541.1630943439.623985/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0906150541.1630943439.623985/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0906150541.1630943439.623985/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0906150541.1630943439.623985/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0906150541.1630943439.623985/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0906150541.1630943439.623985/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210906155039625078-3874'
createTime: '2021-09-06T15:50:44.089267Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-06_08_50_43-11171059842126572900'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0906150541'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-06T15:50:44.089267Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-06_08_50_43-11171059842126572900]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-06_08_50_43-11171059842126572900
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-06_08_50_43-11171059842126572900?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-06_08_50_43-11171059842126572900 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:43.252Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:43.253Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:52.266Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:52.991Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.024Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.090Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.129Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.188Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.227Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.296Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.361Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.410Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.451Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.484Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.547Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.697Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.740Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.771Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.801Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.830Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.860Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.889Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.928Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.954Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:53.992Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:54.018Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:54.051Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:54.079Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:54.134Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:54.180Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:50:54.211Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:51:27.958Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:51:39.126Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:52:10.163Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:52:10.201Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:59:56.864Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:59:56.911Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:59:56.932Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:59:56.967Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-06T15:59:56.992Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-06_08_50_43-11171059842126572900 after 602 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: ad2dcfb2f64e46a485ae9e6da6385a2b and timestamp: 1630944155.680098:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 67
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-06_08_50_43-11171059842126572900?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 20s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/xqqnp7ppjux4y
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #454
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/454/display/redirect?page=changes>
Changes:
[noreply] [BEAM-12769] Adds support for expanding a Java cross-language transform
------------------------------------------
[...truncated 61.09 KB...]
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2539186 sha256=978b1916cdf10b34230aefad01252ce69c52c43bb41804fd6000f69a1d97c53e
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
> Task :runners:google-cloud-dataflow-java:****:compileJava
> Task :sdks:python:apache_beam:testing:load_tests:installGcpTest
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.36 botocore-1.21.36 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :runners:google-cloud-dataflow-java:****:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :runners:google-cloud-dataflow-java:****:classes
> Task :runners:google-cloud-dataflow-java:****:shadowJar
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0905150451.1630857136.243850/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0905150451.1630857136.243850/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0905150451.1630857136.243850/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0905150451.1630857136.243850/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0905150451.1630857136.243850/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0905150451.1630857136.243850/dataflow-****.jar in 2 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0905150451.1630857136.243850/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0905150451.1630857136.243850/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210905155216244877-7087'
createTime: '2021-09-05T15:52:20.335060Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-05_08_52_19-8764731024013804636'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0905150451'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-05T15:52:20.335060Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-05_08_52_19-8764731024013804636]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-05_08_52_19-8764731024013804636
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-05_08_52_19-8764731024013804636?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-05_08_52_19-8764731024013804636 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:19.471Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:19.472Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:26.718Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.308Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.327Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.414Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.447Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.480Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.506Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.557Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.601Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.630Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.647Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.676Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.702Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.724Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.745Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.770Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.799Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.831Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.855Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.879Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.902Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.924Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.953Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:27.979Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:28.019Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:28.048Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:28.095Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:28.129Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:52:28.150Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:53:03.911Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:53:12.970Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:53:42.326Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T15:53:42.354Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T16:01:31.305Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T16:01:31.356Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T16:01:31.391Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T16:01:31.412Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-05T16:01:31.443Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-05_08_52_19-8764731024013804636 after 601 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 8d62f2bb9e0041f98968ef29c11a6cbd and timestamp: 1630857841.8866224:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 65
Traceback (most recent call last):
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-05_08_52_19-8764731024013804636?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 47s
83 actionable tasks: 55 executed, 28 from cache
Publishing build scan...
https://gradle.com/s/yvtyrqkuk5fck
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #453
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/453/display/redirect>
Changes:
------------------------------------------
[...truncated 60.92 KB...]
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2537902 sha256=e59bbf20ee8ef4f38abeb38d3da09b689eb07239bad30739becd2b4557479dd0
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.36 botocore-1.21.36 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0904150506.1630770636.876745/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0904150506.1630770636.876745/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0904150506.1630770636.876745/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0904150506.1630770636.876745/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0904150506.1630770636.876745/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0904150506.1630770636.876745/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0904150506.1630770636.876745/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0904150506.1630770636.876745/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210904155036877851-5691'
createTime: '2021-09-04T15:50:41.366717Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-04_08_50_40-6505150121755224004'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0904150506'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-04T15:50:41.366717Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-04_08_50_40-6505150121755224004]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-04_08_50_40-6505150121755224004
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-04_08_50_40-6505150121755224004?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-04_08_50_40-6505150121755224004 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:40.666Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:40.667Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:48.580Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.324Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.354Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.423Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.447Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.487Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.524Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.597Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.654Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.695Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.725Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.751Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.786Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.814Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.838Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.869Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.904Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.928Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.960Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:49.985Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:50.022Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:50.060Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:50.101Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:50.158Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:50.182Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:50.215Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:50.273Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:50.304Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:50:50.334Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:51:19.523Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:51:40.877Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:52:11.113Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:52:11.140Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:59:54.191Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:59:54.248Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:59:54.276Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:59:54.311Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-04T15:59:54.341Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-04_08_50_40-6505150121755224004 after 602 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 1d35c3a6377c4f859c9ab3ebf51cbd05 and timestamp: 1630771342.8582792:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 82
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-04_08_50_40-6505150121755224004?project=apache-beam-testing
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 8s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/dpxfatgbd3wow
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #452
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/452/display/redirect?page=changes>
Changes:
[Steve Niemitz] [BEAM-12767] Add another unit test for PipelineOptions deserialization
[aydar.zaynutdinov] [BEAM-11873] Add support for writes with returning values in JdbcIO
[Andrew Pilloud] s/org.apache.beam.vendor.calcite.v1_20_0/org.apache.beam.vendor.calcite.v1_26_0/g
[Andrew Pilloud] [BEAM-9379] Update to vendored Calcite to 1.26.0
[Andrew Pilloud] Fix flattened rows
[Andrew Pilloud] Fix DDL
[Andrew Pilloud] Handle BeamRelNode in RelSubset
[Andrew Pilloud] Fix BeamIOPushDown
[Andrew Pilloud] [BEAM-9190] Update BeamBigQuerySqlDialect
[Andrew Pilloud] Remap IN to Search
[Andrew Pilloud] [BEAM-9379] Use byte[] instead of ByteString for (VAR)BINARY in UDFs.
[Andrew Pilloud] [BEAM-9379] Update UDF NULL type mismatch test since there is stricter
[Andrew Pilloud] Fix ZetaSQL window function mapping
[Andrew Pilloud] Fix Bigtable tests that depend on SQL types
[Andrew Pilloud] Workaround CALCITE-4759 in JoinPushThroughJoinRule
[Andrew Pilloud] Disable nested bytes tests, sorry!
[Andrew Pilloud] SqlLine is rotting, Just CAST types for now
[Andrew Pilloud] Update CHANGES.md
[Andrew Pilloud] Up spotbug stack size
[Andrew Pilloud] Fix BeamMatchRel copy
[Andrew Pilloud] partitionKey everywhere
[Andrew Pilloud] Make it functional
[Andrew Pilloud] Update CreateFunction
[Andrew Pilloud] No tpcds dependency
[Andrew Pilloud] Fix default time types
[Ankur Goenka] fixing release date
[Andrew Pilloud] [BEAM-12680] Calcite SqlTransform no longer experimental
[noreply] [BEAM-12823] TestStream Support in Samza Portable Runner (#15421)
------------------------------------------
[...truncated 61.18 KB...]
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<=1.1.2,>=1.0
Using cached google_crc32c-1.1.2-cp37-cp37m-manylinux2014_x86_64.whl (38 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.9.0-py3-none-any.whl (51 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2537902 sha256=85b3f9e4b84d3224ef851ce70a7e032192e56635645afefaeef15e6d16dada4f
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pycparser, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, cffi, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.18.0 azure-storage-blob-12.8.1 boto3-1.18.35 botocore-1.21.35 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.2 google-resumable-media-2.0.2 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.9.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0903150524.1630684241.970832/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0903150524.1630684241.970832/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0903150524.1630684241.970832/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0903150524.1630684241.970832/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0903150524.1630684241.970832/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0903150524.1630684241.970832/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0903150524.1630684241.970832/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0903150524.1630684241.970832/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210903155041972018-7717'
createTime: '2021-09-03T15:50:46.811523Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-03_08_50_45-12769012936603386518'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0903150524'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-03T15:50:46.811523Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-03_08_50_45-12769012936603386518]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-03_08_50_45-12769012936603386518
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-03_08_50_45-12769012936603386518?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-03_08_50_45-12769012936603386518 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:45.970Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:45.971Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:52.191Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:52.980Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.026Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.093Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.129Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.185Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.214Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.284Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.351Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.397Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.431Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.462Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.483Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.515Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.549Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.582Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.609Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.638Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.670Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.707Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.740Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.763Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.807Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.847Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.884Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.917Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:53.975Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:54.018Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:50:54.045Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:51:25.302Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:51:39.182Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:52:10.077Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T15:52:10.115Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T16:00:09.853Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T16:00:09.898Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T16:00:09.924Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T16:00:09.967Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-03T16:00:10.025Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-03_08_50_45-12769012936603386518 after 603 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: e7195ac0ccd042a5ae6c89bbd1ec49dc and timestamp: 1630684965.5497808:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 79
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-03_08_50_45-12769012936603386518?project=apache-beam-testing
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 27s
83 actionable tasks: 53 executed, 30 from cache
Publishing build scan...
https://gradle.com/s/lttunycno2rou
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_PerformanceTests_PubsubIOIT_Python_Streaming #451
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/451/display/redirect?page=changes>
Changes:
[baeminbo] Fix apiclient_test unittest not to fail with no credentials
[noreply] [BEAM-10913] - Adding new Grafana dashboard to monitor GAs post-commit
[aromanenko.dev] Remove a template line
[Steve Niemitz] [BEAM-12767] Fix handling display data in pipeline serialization
[kawaigin] Updated Linux golden screenshots for notebook integration tests.
------------------------------------------
[...truncated 60.88 KB...]
Using cached google_cloud_bigtable-1.2.1-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.2.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.1.0-py2.py3-none-any.whl (234 kB)
Using cached google_cloud_bigtable-1.0.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.34.0-py2.py3-none-any.whl (232 kB)
Using cached google_cloud_bigtable-0.33.0-py2.py3-none-any.whl (230 kB)
Collecting grpc-google-iam-v1<0.12dev,>=0.11.4
Using cached grpc-google-iam-v1-0.11.4.tar.gz (12 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-0.32.2-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.1-py2.py3-none-any.whl (156 kB)
Using cached google_cloud_bigtable-0.32.0-py2.py3-none-any.whl (155 kB)
Using cached google_cloud_bigtable-0.31.1-py2.py3-none-any.whl (154 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
Using cached google_api_core-2.0.0-py2.py3-none-any.whl (92 kB)
Using cached google_api_core-1.31.2-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.1.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (32 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing<3,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.20.0,>=0.8->apache-beam==2.34.0.dev0) (2.4.7)
Collecting pbr>=0.11
Using cached pbr-5.6.0-py2.py3-none-any.whl (111 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (2.1.1)
Collecting attrs>=17.4.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (1.10.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.8.0-py3-none-any.whl (48 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.34.0.dev0) (3.5.0)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)
Collecting wrapt
Using cached wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-5.0.2-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.34.0.dev0-py3-none-any.whl size=2537902 sha256=6b583872f2c68f589365309c69e8e16b43215b6616b5d4c19ad0aaede57362e1
Stored in directory: /home/jenkins/.cache/pip/wheels/ab/30/db/7a1d878abb34081c011192312e662d364ee443467dfb553778
Successfully built apache-beam
Installing collected packages: pyasn1, urllib3, rsa, pyasn1-modules, idna, charset-normalizer, certifi, cachetools, wcwidth, requests, pytz, python-dateutil, pycparser, pluggy, oauthlib, more-itertools, jmespath, googleapis-common-protos, google-auth, attrs, atomicwrites, websocket-client, requests-oauthlib, pytest, numpy, isodate, httplib2, grpcio-gcp, google-crc32c, google-api-core, docopt, cffi, botocore, wrapt, typing-extensions, s3transfer, pytest-forked, pymongo, pydot, pyarrow, proto-plus, pbr, orjson, oauth2client, msrest, hdfs, grpc-google-iam-v1, greenlet, google-resumable-media, google-cloud-core, fasteners, fastavro, execnet, docker, dill, deprecation, cryptography, crcmod, azure-core, avro-python3, testcontainers, tenacity, sqlalchemy, requests-mock, pyyaml, pytest-xdist, pytest-timeout, pyhamcrest, psycopg2-binary, parameterized, pandas, mock, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery, google-apitools, freezegun, boto3, azure-storage-blob, apache-beam
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.34.0.dev0 atomicwrites-1.4.0 attrs-21.2.0 avro-python3-1.9.2.1 azure-core-1.17.0 azure-storage-blob-12.8.1 boto3-1.18.34 botocore-1.21.34 cachetools-4.2.2 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.4 crcmod-1.7 cryptography-3.4.8 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.2 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.4 fasteners-0.16.3 freezegun-1.1.0 google-api-core-1.31.2 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.26.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.1.3 google-resumable-media-2.0.1 googleapis-common-protos-1.53.0 greenlet-1.1.1 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.6.0 httplib2-0.19.1 idna-3.2 isodate-0.6.0 jmespath-0.10.0 mock-2.0.0 more-itertools-8.8.0 msrest-0.6.21 numpy-1.20.3 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.3 pandas-1.2.5 parameterized-0.7.5 pbr-5.6.0 pluggy-0.13.1 proto-plus-1.19.0 psycopg2-binary-2.9.1 pyarrow-4.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.7.2 s3transfer-0.5.0 sqlalchemy-1.4.23 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-3.10.0.2 urllib3-1.26.6 wcwidth-0.2.5 websocket-client-1.2.1 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.34.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20210809" for Docker environment
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0902150509.1630597831.415827/pickled_main_session...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0902150509.1630597831.415827/pickled_main_session in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0902150509.1630597831.415827/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0902150509.1630597831.415827/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0902150509.1630597831.415827/dataflow-****.jar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0902150509.1630597831.415827/dataflow-****.jar in 3 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0902150509.1630597831.415827/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/performance-tests-psio-python-2gb0902150509.1630597831.415827/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--pubsub_namespace_prefix=pubsub_io_performance_']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20210902155031417093-9672'
createTime: '2021-09-02T15:50:36.004008Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-02_08_50_35-15870902107651440599'
location: 'us-central1'
name: 'performance-tests-psio-python-2gb0902150509'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-02T15:50:36.004008Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2021-09-02_08_50_35-15870902107651440599]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2021-09-02_08_50_35-15870902107651440599
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-02_08_50_35-15870902107651440599?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2021-09-02_08_50_35-15870902107651440599 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:35.163Z: JOB_MESSAGE_BASIC: Streaming Engine auto-enabled. Use --experiments=disable_streaming_engine to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:35.164Z: JOB_MESSAGE_BASIC: Dataflow Runner V2 auto-enabled. Use --experiments=disable_runner_v2 to opt out.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:44.765Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.555Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.592Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.659Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.695Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create input/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.725Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.753Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.822Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.889Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.938Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.967Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Split into Create input/Impulse
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:45.995Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/AddRandomKeys into Create input/Split
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.021Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create input/Reshuffle/AddRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.054Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/WriteStream into Create input/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.080Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets into Create input/Reshuffle/ReshufflePerKey/GroupByKey/ReadStream
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.114Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create input/Reshuffle/ReshufflePerKey/GroupByKey/MergeBuckets
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.148Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/Reshuffle/RemoveRandomKeys into Create input/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.182Z: JOB_MESSAGE_DETAILED: Fusing consumer Create input/ReadSplits into Create input/Reshuffle/RemoveRandomKeys
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.229Z: JOB_MESSAGE_DETAILED: Fusing consumer Format to pubsub message in bytes into Create input/ReadSplits
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.263Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time into Format to pubsub message in bytes
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.295Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/ToProtobuf into Measure time
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.326Z: JOB_MESSAGE_DETAILED: Fusing consumer Write to Pubsub/Write into Write to Pubsub/ToProtobuf
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.391Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.420Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.454Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.486Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.546Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.576Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:46.603Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:50:56.283Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:51:29.172Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:51:57.436Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:51:57.472Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:59:51.806Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:59:51.871Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:59:51.903Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:59:51.933Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2021-09-02T15:59:51.959Z: JOB_MESSAGE_BASIC: Stopping **** pool...
WARNING:apache_beam.runners.dataflow.dataflow_runner:Timing out on waiting for job 2021-09-02_08_50_35-15870902107651440599 after 603 seconds
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 2d878dd8bb6b46c1adc21152dbb08f0f and timestamp: 1630598542.8553016:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: pubsub_io_perf_write_runtime Value: 72
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 223, in <module>
PubsubReadPerfTest().run()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 147, in run
self.test()
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/io/gcp/pubsub_io_perf_test.py",> line 183, in test
| 'Write to Pubsub' >> beam.io.WriteToPubSub(self.matcher_topic_name))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-02_08_50_35-15870902107651440599?project=apache-beam-testing
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 1909, in expand
| 'UnKey' >> Map(lambda k_v: k_v[1]))
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 642, in apply
transform.transform, pvalueish, label or transform.label)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 652, in apply
return self.apply(transform, pvalueish)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 215, in apply_PTransform
return transform.expand(input)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2048, in expand
return pcoll | GroupByKey() | 'Combine' >> CombineValues(
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pvalue.py",> line 137, in __or__
return self.pipeline.apply(ptransform, self)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/pipeline.py",> line 698, in apply
pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 141, in apply
return super(DataflowRunner, self).apply(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/runner.py",> line 185, in apply
return m(transform, input, options)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 826, in apply_GroupByKey
return transform.expand(pcoll)
File "<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2349, in expand
raise ValueError(msg)
ValueError: GroupByKey: Unsafe trigger: `Repeatedly(AfterCount(2097152))` may lose data. Reason: CONDITION_NOT_GUARANTEED. This can be overriden with the --allow_unsafe_triggers flag.
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PerformanceTests_PubsubIOIT_Python_Streaming/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 4s
83 actionable tasks: 54 executed, 29 from cache
Publishing build scan...
https://gradle.com/s/p2fhirdre5b6g
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org