You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2020/07/31 14:50:20 UTC
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #419
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/419/display/redirect?page=changes>
Changes:
[Robert Bradshaw] Convert katas to use with syntax rather than explicit run call.
[Pablo Estrada] Standardizing BigQuery job names in Beam Python and Java SDKs
[Pablo Estrada] Naming BQ Copy jobs consistently in Python
[Robert Bradshaw] Move class definition outside of with statement.
[ningk] [BEAM-10545] KernelModel and jest tests
[Robert Bradshaw] Fix placeholder offsets.
[douglas.damon] Add Composite transforms to Go SDK katas
[jiyongjung] Bump google cloud bigquery to 1.26.1
[Maximilian Michels] [BEAM-10602] Display Python streaming metrics in Grafana dashboard
[Rui Wang] remove redundent precommits.
[ningk] Change the syntax of private _onIOPub to a function declaration instead
[douglas.damon] Update stepik
[Kenneth Knowles] Upgrade spotbugs-gradle-plugin to 3.0.0
[Kenneth Knowles] Upgrade Spotbugs from 3.1.12 to 4.0.6
[simonepri] Add failing test for count on an empty pcollection
[simonepri] Fix go count on an empty pcollection
[noreply] [BEAM-10331] Add SnowflakeIO to list of built-in IO transforms. (#12099)
[Robert Burke] [BEAM-10610] Add ExternalWorker server
[Robert Burke] [BEAM-10610] Support loopback in universal
[Robert Burke] [BEAM-10610] Set GRPC recv limit to max.
[Robert Burke] [BEAM-10610] Support external env cfg.
[Robert Burke] [BEAM-10610] Make missed logs more readable.
[Robert Burke] [BEAM-10610] Fix control response race condition.
[Pablo Estrada] Applying fixes from comments.
[noreply] [BEAM-10559] Add apache_beam.examples.sql_taxi (#12399)
------------------------------------------
[...truncated 27.32 KB...]
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<=0.13.0,>=0.12.0
Using cached google_cloud_dlp-0.13.0-py2.py3-none-any.whl (151 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<1.14.0,>=1.8.0
Using cached google_cloud_videointelligence-1.13.0-py2.py3-none-any.whl (177 kB)
Collecting google-cloud-vision<0.43.0,>=0.38.0
Using cached google_cloud_vision-0.42.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting botocore<1.18.0,>=1.17.32
Using cached botocore-1.17.32-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.6.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.0-py2.py3-none-any.whl (42 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-auth<2.0dev,>=1.19.1
Using cached google_auth-1.20.0-py2.py3-none-any.whl (91 kB)
Collecting google-crc32c>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2127447 sha256=aed55e39335fe2f9b5d5eeb0cbec7b47e8fd8a30425161b401fcd903f16c2733
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, certifi, urllib3, chardet, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, googleapis-common-protos, google-auth, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, packaging, attrs, atomicwrites, more-itertools, wcwidth, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, websocket-client, docker, deprecation, blindspin, colorama, crayons, wrapt, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.32 botocore-1.17.32 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.0 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.0.0 google-cloud-core-1.3.0 google-cloud-datastore-1.7.4 google-cloud-dlp-0.13.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.0.2 google-cloud-spanner-1.13.0 google-cloud-videointelligence-1.13.0 google-cloud-vision-0.42.0 google-crc32c-0.1.0 google-resumable-media-0.7.0 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0731125347.1596206236.108656/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0731125347.1596206236.108656/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0731125347.1596206236.108656/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0731125347.1596206236.108656/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-07-31T14:37:17.898232Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-07-31_07_37_16-5363261014784909503'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0731125347'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-07-31T14:37:17.898232Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-07-31_07_37_16-5363261014784909503]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-07-31_07_37_16-5363261014784909503
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-07-31_07_37_16-5363261014784909503?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-07-31_07_37_16-5363261014784909503 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:20.437Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:21.677Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:21.742Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:21.781Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:21.810Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:21.873Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:21.915Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:21.950Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:21.985Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.023Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.057Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.091Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.116Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.150Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.184Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.219Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.264Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.301Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.335Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.474Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.681Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.728Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.753Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.790Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.854Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:22.920Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-07-31_07_37_16-5363261014784909503 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:33.805Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:47.079Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:47.117Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:37:52.540Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:39:29.662Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:39:29.701Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:47:03.826Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:47:03.948Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:47:04.005Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:47:04.064Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:49:08.525Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:49:08.601Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:49:08.729Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:49:08.786Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:49:08.819Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:50:06.290Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:50:06.329Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-07-31T14:50:06.368Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-07-31_07_37_16-5363261014784909503 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: d18774576892458cb88d487df6e3813e and timestamp: 1596207018.69823:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 385
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224968243803228
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 2s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/okwq222rtzyv6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal :
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #445
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/445/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #444
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/444/display/redirect?page=changes>
Changes:
[Udi Meiri] Silence red PYTHONPATH warning from tox
[Robin Qiu] Propagate BigQuery streaming insert throttled time to Dataflow worker in
[Kamil Wasilewski] [BEAM-10524] JSON decoder for ReadFromBigQuery supports repeatable
[noreply] Merge pull request #12415 from [BEAM-10603] Add the RecordingManager and
[noreply] [BEAM-10777] Add two blog posts detailing changes to the type hints
------------------------------------------
[...truncated 27.33 KB...]
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting botocore<1.18.0,>=1.17.48
Using cached botocore-1.17.48-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam, boto3
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2204369 sha256=0d05dcaa61c6d4bc8c03cbd666b1c26c677b0aa37dda91363932f08121b8169a
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Building wheel for boto3 (setup.py): started
Building wheel for boto3 (setup.py): finished with status 'done'
Created wheel for boto3: filename=boto3-1.14.48-py2.py3-none-any.whl size=127852 sha256=2163e9b0d03fec61a7fcf4c46fe818a4f7f6e9cc596f8aea33e197fcb64c48ac
Stored in directory: /home/jenkins/.cache/pip/wheels/b0/f8/d9/d2a816eb3b5fed353cf3e7746d2c3bd0cac694045d063a2b53
Successfully built apache-beam boto3
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, certifi, chardet, urllib3, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, more-itertools, attrs, wcwidth, packaging, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, deprecation, websocket-client, docker, colorama, crayons, blindspin, wrapt, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.48 botocore-1.17.48 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.3 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0825125343.1598366237.244709/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0825125343.1598366237.244709/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0825125343.1598366237.244709/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0825125343.1598366237.244709/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-25T14:37:19.005549Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-25_07_37_17-11432471775787433052'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0825125343'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-25T14:37:19.005549Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-25_07_37_17-11432471775787433052]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-25_07_37_17-11432471775787433052
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-25_07_37_17-11432471775787433052?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-25_07_37_17-11432471775787433052 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:21.779Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:22.728Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:22.765Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:22.817Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:22.942Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.025Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.062Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.098Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.135Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.183Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.216Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.353Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.381Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.407Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.433Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.470Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.502Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.533Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.619Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.846Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.917Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.956Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.980Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:24.012Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:24.075Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:24.135Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-25_07_37_17-11432471775787433052 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:49.553Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:53.072Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:53.107Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:58.418Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:39:31.637Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:39:31.663Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:46:59.006Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:46:59.099Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:46:59.142Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:46:59.196Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.042Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.205Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.284Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.419Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.450Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:57.995Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:58.029Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:58.063Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-25_07_37_17-11432471775787433052 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 9eb2c8fb6b5c433886fc2f7c094d8de3 and timestamp: 1598367014.8794644:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 372
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224970403798592
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224970403676592
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 57s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/mnx7ntwhfumsq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #443
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/443/display/redirect?page=changes>
Changes:
[tobiasz.kedzierski] [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac
------------------------------------------
[...truncated 27.03 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting botocore<1.18.0,>=1.17.47
Using cached botocore-1.17.47-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2197167 sha256=76d3fcb367bf0b96464e2287ee013686707bcd6ae1e8d8693e4bba977f76ee8c
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, urllib3, chardet, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, attrs, packaging, atomicwrites, more-itertools, wcwidth, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, deprecation, wrapt, websocket-client, docker, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.47 botocore-1.17.47 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.3 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0824125344.1598279836.241577/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0824125344.1598279836.241577/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0824125344.1598279836.241577/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0824125344.1598279836.241577/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-24T14:37:17.876845Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-24_07_37_16-3221239520918555375'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0824125344'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-24T14:37:17.876845Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-24_07_37_16-3221239520918555375]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-24_07_37_16-3221239520918555375
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-24_07_37_16-3221239520918555375?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-24_07_37_16-3221239520918555375 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:20.751Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.675Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.701Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.759Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.787Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.833Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.874Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.900Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.927Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.947Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.973Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.999Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.021Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.049Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.069Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.098Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.122Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.149Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.185Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.355Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.413Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.453Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.473Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.507Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.563Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.627Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-24_07_37_16-3221239520918555375 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:37.104Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:48.882Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:39:26.056Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:39:26.078Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:46:50.721Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:46:50.825Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:46:50.912Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:46:50.981Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:48.854Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:48.909Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:48.989Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:49.055Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:49.082Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:49:48.572Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:49:48.644Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:49:48.674Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-24_07_37_16-3221239520918555375 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: bfbcd0211acd4f4094c2966889476c50 and timestamp: 1598280603.722677:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 367
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224970317383426
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224970317268208
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 46s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/nz7aqj342ggss
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #442
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/442/display/redirect?page=changes>
Changes:
[noreply] [BEAM-7587] Spark portable streaming (#12157)
[noreply] Merge pull request #12653 from [BEAM-10758] Centralized Glob Translation
------------------------------------------
[...truncated 26.97 KB...]
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.47
Using cached botocore-1.17.47-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2197167 sha256=a887d1b2964f58393c4d797c4a3fff7262d8b8cb113ca677464017d0d29800a7
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, certifi, idna, chardet, urllib3, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, attrs, more-itertools, wcwidth, packaging, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, deprecation, blindspin, colorama, crayons, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.47 botocore-1.17.47 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0823125359.1598193436.224736/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0823125359.1598193436.224736/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0823125359.1598193436.224736/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0823125359.1598193436.224736/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-23T14:37:17.721574Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-23_07_37_16-5148909312103907913'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0823125359'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-23T14:37:17.721574Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-23_07_37_16-5148909312103907913]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-23_07_37_16-5148909312103907913
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-23_07_37_16-5148909312103907913?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-23_07_37_16-5148909312103907913 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:20.227Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.047Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.079Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.114Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.134Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.203Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.234Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.266Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.295Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.329Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.363Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.398Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.430Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.465Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.498Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.532Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.566Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.601Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.635Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.780Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.841Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.885Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.917Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.960Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:22.042Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:22.102Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-23_07_37_16-5148909312103907913 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:38.482Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:50.738Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:39:34.534Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:39:34.565Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:46:50.635Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:46:50.700Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:46:50.793Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:46:50.858Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.543Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.612Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.672Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.730Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.767Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:49:41.253Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:49:41.298Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:49:41.336Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-23_07_37_16-5148909312103907913 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 73d874d3b3c44293af7eb47ac6675ad7 and timestamp: 1598194193.5487041:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 359
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224970230977252
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 36s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/73jgdqpzxlxkm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #441
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/441/display/redirect?page=changes>
Changes:
[Udi Meiri] Color Jenkins logs support
[ningk] Added a whitespace lint precommit job.
[ningk] Removed trailing whitespaces from all markdown and gradle build files.
[ningk] Updated jenkins precommit job README, added comments of what needs to be
[ningk] Removed trailing whitespaces from HEAD changes
[Alan Myrvold] [BEAM-10049] Add licenses for go dependencies in python, java, and go
[Robin Qiu] Fix bug in JoinScanWithRefConverter
[noreply] Propagate BigQuery streaming insert throttled time to Dataflow worker in
[tysonjh] [BEAM-10751] Revert "Extending archiveJunit post-commit task with
[noreply] [BEAM-10764] Make is_in_ipython robust (#12641)
[noreply] [BEAM-10766] Fix flake where ByteString coercion could modify cached
[ningk] [BEAM-10771] Added Whitespacelint job to pull request template
[noreply] Use setCoder, not setSchema (#12662)
[noreply] [BEAM-10703] Prepare Dataflow Java runner for shardable states (#12578)
[noreply] [BEAM-10549] Improve runtime type checking performance for the Python
[Kyle Weaver] [BEAM-10460] Remove SparkPortableExecutionTest altogether.
[noreply] [BEAM-10788] Also disable immutability checking for transforms within
[noreply] [BEAM-5715, BEAM-8862, BEAM-8702] Removes all references to grpc_all and
[noreply] [BEAM-10761] Prefer TestPubsub#assertSubscriptionEventuallyExists when
[noreply] [BEAM-10654] Implemented ExternalSchemaIOTransformRegistrar for jdbc
[noreply] Merge pull request #12580 from [BEAM-2855] nexmark python suite
[noreply] [BEAM-9918] Adding tests and documentation to xlang components (#12667)
------------------------------------------
[...truncated 26.89 KB...]
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.47
Using cached botocore-1.17.47-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2197167 sha256=c779d8989b9142583c7827787c5ca6f585355a78d87479168bea25986b261bc4
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, urllib3, idna, chardet, certifi, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, attrs, wcwidth, atomicwrites, packaging, more-itertools, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, wrapt, websocket-client, docker, deprecation, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.47 botocore-1.17.47 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0822125344.1598107038.368177/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0822125344.1598107038.368177/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0822125344.1598107038.368177/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0822125344.1598107038.368177/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-22T14:37:19.972862Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-22_07_37_18-7377215218196749982'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0822125344'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-22T14:37:19.972862Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-22_07_37_18-7377215218196749982]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-22_07_37_18-7377215218196749982
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-22_07_37_18-7377215218196749982?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-22_07_37_18-7377215218196749982 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:22.496Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.369Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.406Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.443Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.478Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.528Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.562Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.592Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.622Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.651Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.683Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.719Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.750Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.782Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.817Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.844Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.879Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.906Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.937Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.070Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.139Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.181Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.207Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.246Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.289Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.351Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-22_07_37_18-7377215218196749982 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:57.238Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:38:00.145Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:39:36.809Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:39:36.836Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:47:02.455Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:47:02.589Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:47:02.644Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:47:02.716Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.004Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.067Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.140Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.195Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.231Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:50:02.062Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:50:02.094Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:50:02.132Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-22_07_37_18-7377215218196749982 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: edc65f7ce06c43a194f0c712a2ea5f2d and timestamp: 1598107815.9023364:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 372
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 55s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/wk7hddcz7lcf4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #440
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/440/display/redirect?page=changes>
Changes:
[david.janicek] [BEAM-10292] LocalFileSystem.matchNewResource method throws
[Kamil Wasilewski] [BEAM-10674] Python CoGBK: Remove unnecessary step. Adjust parameters to
[Kamil Wasilewski] [BEAM-10674] Python CoGBK: Add streaming job
[Kyle Weaver] [BEAM-9118] Increase portable_runner_test timeout.
[Kyle Weaver] [BEAM-10770] Remove DataflowPortabilityApiUnsupported annotation.
[daniel.o.programmer] [BEAM-3301] Consolidating starcgen method names in graph/fn.go and
[daniel.o.programmer] [BEAM-10289] Fixing miscalculated SDF split fraction.
[david.janicek] [BEAM-10292] changes after review
[noreply] [BEAM-10701] Generate Codecov XML (#12566)
[noreply] [BEAM-9979] Fix read index being reported to be reset after any metrics
[noreply] [BEAM-9546] DataframeTransform can now consume a schema-aware
------------------------------------------
[...truncated 26.89 KB...]
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting botocore<1.18.0,>=1.17.47
Using cached botocore-1.17.47-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2179154 sha256=38decee90a3737b5fd7c8ce126bf38375622c2629a7099ebab81421fcdff84ac
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, urllib3, chardet, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, wcwidth, attrs, packaging, more-itertools, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, deprecation, blindspin, websocket-client, docker, wrapt, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.47 botocore-1.17.47 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0821125344.1598020635.025780/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0821125344.1598020635.025780/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0821125344.1598020635.025780/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0821125344.1598020635.025780/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-21T14:37:16.559357Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-21_07_37_15-11450326444839000613'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0821125344'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-21T14:37:16.559357Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-21_07_37_15-11450326444839000613]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-21_07_37_15-11450326444839000613
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-21_07_37_15-11450326444839000613?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-21_07_37_15-11450326444839000613 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:19.448Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.256Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.300Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.359Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.402Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.467Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.503Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.531Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.569Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.603Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.636Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.658Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.690Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.781Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.851Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.914Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.951Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.993Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.029Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.301Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.370Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.418Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.442Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.477Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.541Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.596Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-21_07_37_15-11450326444839000613 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:37.677Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:47.424Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:47.447Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:38:29.807Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:39:27.986Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:39:28.017Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:47:02.002Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:47:02.081Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:47:02.130Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:47:02.211Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:03.762Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:03.873Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:03.948Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:04.014Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:04.044Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:54.491Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:54.541Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:54.597Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-21_07_37_15-11450326444839000613 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 8b8ed40169604de388612ddb1545aac5 and timestamp: 1598021407.4029233:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 376
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224970058198318
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224970058079616
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 50s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/aoebgsxlrjxyu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #439
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/439/display/redirect?page=changes>
Changes:
[chamikaramj] Include coders embedded in CombinePayload when converting x-lang
[zyichi] Allow Nexmark launcher to publish human-readable events to pubsub.
[douglas.damon] Add Filter lesson to Go SDK katas
[douglas.damon] Reformat code for readability
[yoshiki.obata] [BEAM-9980] Configure Python versions for direct test suite tasks via
[irvi.fa] [BEAM-10753] Add Slack link invitation on README
[Luke Cwik] [BEAM-10756] Fix empty pull response to not ack and to not throw
[ettarapp] clarifying unclear comments
[noreply] Update README.md
[noreply] [BEAM-3301] Adding SDF Go Dataflow translation. (#12629)
[kevinsijo] [BEAM-9919] Refining xlang types and wrappers
[noreply] [BEAM-10752] Use TestPubsubSignal in PubsubToBigqueryIT (#12625)
[douglas.damon] Change license info format in task-info.yaml
[douglas.damon] Update stepik course
[Luke Cwik] [BEAM-9979] Support a reset call that happens after the bundle is done
[noreply] [BEAM-10200] Add optional experiment to enable heap dump through the …
[kevinsijo] [BEAM-9919] xlang transforms wrapped with namespaces
[kevinsijo] [BEAM-9919] cleaning namespace.go
[kevinsijo] [BEAM-9920] Enabling artifact staging for xlang transforms
[kevinsijo] documentation(beam/sdks/go): added apache license and function docstring
[noreply] [BEAM-9547] Implement dataframes top, join, merge. (#12516)
[kevinsijo] fix(beam/sdks/go): correcting changed function name
[noreply] Merge pull request #12581 from [BEAM-10378] Add Azure Blob Storage
------------------------------------------
[...truncated 26.72 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.46
Using cached botocore-1.17.46-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2174879 sha256=97a826fe137a8164f736f998d2854ed048395883171070ae54b0e868e3454aca
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, chardet, urllib3, certifi, idna, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, wcwidth, atomicwrites, packaging, attrs, more-itertools, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, deprecation, wrapt, colorama, crayons, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.46 botocore-1.17.46 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0820125344.1597934236.608705/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0820125344.1597934236.608705/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0820125344.1597934236.608705/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0820125344.1597934236.608705/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-20T14:37:18.196273Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-20_07_37_17-17211290841274263071'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0820125344'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-20T14:37:18.196273Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-20_07_37_17-17211290841274263071]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-20_07_37_17-17211290841274263071
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-20_07_37_17-17211290841274263071?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-20_07_37_17-17211290841274263071 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:20.687Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.671Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.701Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.728Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.758Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.817Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.860Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.890Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.918Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.956Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.995Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.034Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.067Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.092Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.122Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.158Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.205Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.243Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.273Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.395Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.453Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.501Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.525Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.591Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.660Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.734Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-20_07_37_17-17211290841274263071 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:33.311Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:50.287Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:39:32.776Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:39:32.815Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:46:55.235Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:46:55.296Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:46:55.415Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:46:55.504Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.252Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.319Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.387Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.432Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.455Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:49:48.696Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:49:48.742Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:49:48.768Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-20_07_37_17-17211290841274263071 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 798a6aff9d484bb391028126939e25da and timestamp: 1597934999.0144124:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 366
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969971790948
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969971672816
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 42s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/inmmtd2jqtwxy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #438
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/438/display/redirect?page=changes>
Changes:
[Kyle Weaver] [BEAM-9956] Remove unbalanced code markup.
[zyichi] Bump up advanceProcessingTime duration in ParDoTest
[tobiasz.kedzierski] [BEAM-10623] Add workflow to run python tests on Linux/Windows/Mac
[tobiasz.kedzierski] [BEAM-10624] dtype explicit for the numpy arrays
[noreply] [BEAM-10715] Update Jet Runner validates runner testing (#12567)
[noreply] [BEAM-10557] Implemented SchemaIOProvider for DataStoreV1, Refactored
[Steve Niemitz] [BEAM-10523] Add support for custom DatumWriters to AvroIO.Write
[Luke Cwik] [BEAM-10040, BEAM-6804] Increase wait times to reduce flakiness.
[noreply] [BEAM-10752] add use_deprecated_read experiment for testSimpleInsert,
[ekirpichov] Add enum34 to manual_licenses
[tobiasz.kedzierski] [BEAM-BEAM-10667] Make github actions bulding wheels bucket name stored
[noreply] Merge pull request #12597: [BEAM-10685] Added integration test for
------------------------------------------
[...truncated 26.74 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting botocore<1.18.0,>=1.17.45
Using cached botocore-1.17.45-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172826 sha256=68bd9d4a99b4983341eadc3d4077e7c78c250a380ab28047bc2abb0b1dc4352c
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, chardet, idna, urllib3, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, attrs, wcwidth, more-itertools, packaging, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, wrapt, websocket-client, docker, colorama, crayons, deprecation, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.45 botocore-1.17.45 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0819125407.1597847847.824123/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0819125407.1597847847.824123/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0819125407.1597847847.824123/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0819125407.1597847847.824123/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-19T14:37:29.501334Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-19_07_37_28-436199677609419665'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0819125407'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-19T14:37:29.501334Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-19_07_37_28-436199677609419665]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-19_07_37_28-436199677609419665
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-19_07_37_28-436199677609419665?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-19_07_37_28-436199677609419665 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:31.830Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.593Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.623Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.668Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.696Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.761Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.798Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.826Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.853Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.888Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.919Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.950Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.979Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.012Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.047Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.085Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.119Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.147Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.182Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.382Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.448Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.487Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.507Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.546Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.608Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.678Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-19_07_37_28-436199677609419665 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:38:00.262Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:38:08.473Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:39:44.033Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:39:44.074Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:47:16.440Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:47:16.573Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:47:16.612Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:47:16.685Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.234Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.297Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.380Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.434Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.461Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:50:11.129Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:50:11.160Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:50:11.188Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-19_07_37_28-436199677609419665 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 5c9423c3789a47ecab88e533d660fde7 and timestamp: 1597848620.3681748:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 373
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969885411772
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969885294068
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 3s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/ua5rjgl6p36u2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #437
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/437/display/redirect?page=changes>
Changes:
[yoshiki.obata] [BEAM-9980] add groovy functions for python versions
[yoshiki.obata] [BEAM-9980] update dataflow test-suites to switch python versions using
[heejong] [BEAM-10679] improving XLang KafkaIO streaming test
[heejong] fix formatting
[heejong] fix incorrect coder issue
[heejong] use Create
[yoshiki.obata] [BEAM-9980] version switchable dataflow tasks to be invoked
[yoshiki.obata] [BEAM-9980] :sdks:python:test-suites:dataflow included in
[tobiasz.kedzierski] Improve CI documentation
[Luke Cwik] [BEAM-10670] Make key coder deterministic by using upstream PCollection
[Udi Meiri] [BEAM-10697] Remove testPy2Cython from precommit
[noreply] [BEAM-9891] Generate query execution summary table after finishing jobs
[noreply] [BEAM-9919] Added an External Transform API to Go SDK (#12445)
------------------------------------------
[...truncated 24.38 KB...]
Collecting mock<3.0.0,>=1.0.1
Using cached mock-2.0.0-py2.py3-none-any.whl (56 kB)
Collecting numpy<2,>=1.14.3
Using cached numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)
Collecting pymongo<4.0.0,>=3.8.0
Using cached pymongo-3.11.0-cp37-cp37m-manylinux2014_x86_64.whl (511 kB)
Processing /home/jenkins/.cache/pip/wheels/86/73/7a/3b3f76a2142176605ff38fbca574327962c71e25a43197a4c1/oauth2client-3.0.0-py3-none-any.whl
Requirement already satisfied: protobuf<4,>=3.12.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from apache-beam==2.25.0.dev0) (3.13.0)
Collecting pydot<2,>=1.2.0
Using cached pydot-1.4.1-py2.py3-none-any.whl (19 kB)
Collecting python-dateutil<3,>=2.8.0
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz>=2018.3
Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting requests<3.0.0,>=2.24.0
Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting typing-extensions<3.8.0,>=3.7.0
Using cached typing_extensions-3.7.4.2-py3-none-any.whl (22 kB)
Processing /home/jenkins/.cache/pip/wheels/bc/49/5f/fdb5b9d85055c478213e0158ac122b596816149a02d82e0ab1/avro_python3-1.9.2.1-py3-none-any.whl
Collecting pyarrow<0.18.0,>=0.15.1
Using cached pyarrow-0.17.1-cp37-cp37m-manylinux2014_x86_64.whl (63.8 MB)
Collecting boto3>=1.9
Using cached boto3-1.14.44-py2.py3-none-any.whl (129 kB)
Collecting cachetools<4,>=3.1.0
Using cached cachetools-3.1.1-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/19/b5/2f/1cc3cf2b31e7a9cd1508731212526d9550271274d351c96f16/google_apitools-0.5.31-py3-none-any.whl
Collecting google-auth<2,>=1.18.0
Using cached google_auth-1.20.1-py2.py3-none-any.whl (91 kB)
Collecting google-cloud-datastore<2,>=1.7.1
Using cached google_cloud_datastore-1.15.0-py2.py3-none-any.whl (132 kB)
Collecting google-cloud-pubsub<2,>=0.39.0
Using cached google_cloud_pubsub-1.7.0-py2.py3-none-any.whl (144 kB)
Collecting google-cloud-bigquery<2,>=1.6.0
Using cached google_cloud_bigquery-1.26.1-py2.py3-none-any.whl (170 kB)
Collecting google-cloud-core<2,>=0.28.1
Using cached google_cloud_core-1.4.1-py2.py3-none-any.whl (26 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting botocore<1.18.0,>=1.17.44
Using cached botocore-1.17.44-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172471 sha256=aa380437a69d7ed97add617c86e0a607bb166abd7088ed7d58a902181d7c3b36
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, urllib3, chardet, certifi, idna, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, attrs, more-itertools, packaging, wcwidth, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, colorama, crayons, websocket-client, docker, blindspin, wrapt, deprecation, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.44 botocore-1.17.44 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0818141653.1597761439.456857/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0818141653.1597761439.456857/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0818141653.1597761439.456857/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0818141653.1597761439.456857/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--iterations=4', '--fanout=1']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--iterations=4', '--fanout=1']
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 150, in run
self.result = self.pipeline.run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/test_pipeline.py",> line 112, in run
False if self.not_use_test_runner_api else test_runner_api))
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/pipeline.py",> line 521, in run
allow_proto_holders=True).run(False)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/pipeline.py",> line 534, in run
return self.runner.run_pipeline(self, self._options)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 592, in run_pipeline
self.dataflow_client.create_job(self.job), self)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/utils/retry.py",> line 236, in wrapper
return fun(*args, **kwargs)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",> line 681, in create_job
return self.submit_job_description(job)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/utils/retry.py",> line 236, in wrapper
return fun(*args, **kwargs)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",> line 748, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",> line 667, in Create
config, request, global_params=global_params)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages/apitools/base/py/base_api.py",> line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages/apitools/base/py/base_api.py",> line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages/apitools/base/py/base_api.py",> line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpConflictError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Tue, 18 Aug 2020 14:37:20 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '409', 'content-length': '334', '-content-encoding': 'gzip'}>, content <{
"error": {
"code": 409,
"message": "(569405d57578a32d): The workflow could not be created. Causes: (74dcf4736e5d95f0): There is already an active job named load-tests-python-dataflow-batch-gbk-6-0818141653. If you want to submit a second job, try again by setting a different name.",
"status": "ALREADY_EXISTS"
}
}
>
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1m 3s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/g6wqoojyi5q6q
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #436
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/436/display/redirect?page=changes>
Changes:
[katarzyna.kucharczyk] [BEAM-10672] Added streaming option to combine test
[katarzyna.kucharczyk] [BEAM-10672] Added experimental dataflow param
[katarzyna.kucharczyk] [BEAM-10672] Spottless Apply
[katarzyna.kucharczyk] [BEAM-10672] Updated streaming commands
[katarzyna.kucharczyk] [BEAM-10672] Fixes after review in combine python load test
[katarzyna.kucharczyk] [BEAM-10672] Changed jobType variable into mode to declare if it is
------------------------------------------
[...truncated 26.88 KB...]
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting botocore<1.18.0,>=1.17.43
Using cached botocore-1.17.43-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172406 sha256=7f4bbd3f156da8ad4398f14f2560722eca1378e5bbb72a634fce431879f96465
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, certifi, urllib3, chardet, idna, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, packaging, wcwidth, attrs, more-itertools, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, deprecation, websocket-client, docker, wrapt, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.43 botocore-1.17.43 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0817125556.1597675048.243259/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0817125556.1597675048.243259/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0817125556.1597675048.243259/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0817125556.1597675048.243259/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-17T14:37:29.888404Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-17_07_37_28-2818844421715832941'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0817125556'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-17T14:37:29.888404Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-17_07_37_28-2818844421715832941]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-17_07_37_28-2818844421715832941
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-17_07_37_28-2818844421715832941?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-17_07_37_28-2818844421715832941 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:32.655Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.520Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.557Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.594Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.623Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.714Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.758Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.794Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.817Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.851Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.884Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.910Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.944Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.979Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.029Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.069Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.103Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.134Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.172Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.301Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.367Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.416Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.451Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.495Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.556Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.625Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-17_07_37_28-2818844421715832941 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:38:01.777Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:38:01.813Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:38:04.107Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:38:07.184Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:39:39.058Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:39:39.085Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:47:15.734Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:47:15.783Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:47:15.820Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:47:15.897Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.605Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.688Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.754Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.802Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.830Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:50:03.569Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:50:03.603Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:50:03.637Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-17_07_37_28-2818844421715832941 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 1284ec679e944bcab07b3610e7c79db7 and timestamp: 1597675815.6649678:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 366
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969712605214
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969712493324
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 53s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/qjspdkfrricgc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #435
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/435/display/redirect?page=changes>
Changes:
[Boyuan Zhang] Remove redundant setMaxNumRecords and consumerFactoryFn.
[Luke Cwik] [BEAM-10670] Make Read execute as a splittable DoFn by default for the
[noreply] Merge pull request #12575: [BEAM-10707] Adding Python docs precommit as
------------------------------------------
[...truncated 26.65 KB...]
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting botocore<1.18.0,>=1.17.43
Using cached botocore-1.17.43-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172406 sha256=0057d3a57236d954c1f6a388cc2f7fb9944cd9f2a1a2abe403d030017467df00
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, urllib3, idna, chardet, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, more-itertools, packaging, attrs, atomicwrites, wcwidth, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, blindspin, colorama, crayons, websocket-client, docker, deprecation, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.43 botocore-1.17.43 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0816125613.1597588647.925574/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0816125613.1597588647.925574/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0816125613.1597588647.925574/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0816125613.1597588647.925574/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-16T14:37:29.630461Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-16_07_37_28-6918673436260540560'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0816125613'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-16T14:37:29.630461Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-16_07_37_28-6918673436260540560]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-16_07_37_28-6918673436260540560
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-16_07_37_28-6918673436260540560?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-16_07_37_28-6918673436260540560 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:32.161Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:32.934Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:32.968Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.003Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.036Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.094Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.125Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.157Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.189Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.212Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.248Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.279Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.305Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.325Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.347Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.382Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.416Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.450Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.475Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.633Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.692Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.737Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.771Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.812Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.860Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.916Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-16_07_37_28-6918673436260540560 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:58.866Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:38:04.664Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:39:41.884Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:39:41.915Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:47:04.103Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:47:04.185Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:47:04.268Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:47:04.321Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.567Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.628Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.705Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.770Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.802Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:50:04.731Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:50:04.767Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:50:04.795Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-16_07_37_28-6918673436260540560 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 20aa44a18907477294df4d938d1d0745 and timestamp: 1597589415.4301322:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 366
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 55s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/x2krlh44thuxw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #434
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/434/display/redirect?page=changes>
Changes:
[Luke Cwik] [BEAM-10688] Euphoria assumes that all type descriptors are resolvable
[Luke Cwik] [BEAM-10670] Use fraction of remainder if consumed fraction is unknown
[Luke Cwik] [BEAM-10670] Improve splitting logic to prefer splits upto the the
[Luke Cwik] [BEAM-10670] Fix passing forward the self-checkpoint from the
[Boyuan Zhang] Enable dataflow streaming engine when running runner_v2 and streaming.
[Boyuan Zhang] Fix formatter.
[Boyuan Zhang] Use unbounded wrapper for Kafka Read.
[je.ik] [BEAM-10691] Use FlinkStateInternals#addWatermarkHoldUsage for timer
[noreply] [BEAM-9615] Map user types to Schema reps. (#12554)
[noreply] fixed a typo in S3TestUtils (#12582)
[neville.lyh] [BEAM-10612] Add flink 1.11 runner
[Boyuan Zhang] Scale progress with respect to windows observation.
[noreply] [BEAM-9547] Implement some methods for deferred Series. (#12534)
[Robert Burke] Fix broken build.
[Luke Cwik] [BEAM-8025] Update tests to use TemporaryFolder instead of rolling their
------------------------------------------
[...truncated 26.93 KB...]
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting botocore<1.18.0,>=1.17.43
Using cached botocore-1.17.43-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam, boto3
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172406 sha256=7714d31a5149788b6cea204fd12ba1f12209a8714bf9fe57db5b610bcdb6d9fd
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Building wheel for boto3 (setup.py): started
Building wheel for boto3 (setup.py): finished with status 'done'
Created wheel for boto3: filename=boto3-1.14.43-py2.py3-none-any.whl size=127852 sha256=683cbfa20b1dce577c795ed52d32340a84770e610811ea746df0bad2db15da10
Stored in directory: /home/jenkins/.cache/pip/wheels/2a/2a/a0/48b976bfdd536c39baf38c9f11206ccaad257284eb7cd4aeb2
Successfully built apache-beam boto3
Installing collected packages: crcmod, dill, pytz, fastavro, certifi, urllib3, chardet, idna, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, wcwidth, atomicwrites, more-itertools, attrs, packaging, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, websocket-client, docker, blindspin, deprecation, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.43 botocore-1.17.43 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0815125556.1597502249.024011/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0815125556.1597502249.024011/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0815125556.1597502249.024011/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0815125556.1597502249.024011/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-15T14:37:30.711981Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-15_07_37_29-7011471637501784593'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0815125556'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-15T14:37:30.711981Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-15_07_37_29-7011471637501784593]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-15_07_37_29-7011471637501784593
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-15_07_37_29-7011471637501784593?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-15_07_37_29-7011471637501784593 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:33.261Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:33.975Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.012Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.054Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.095Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.168Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.213Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.252Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.288Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.321Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.355Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.392Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.427Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.464Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.500Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.539Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.572Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.620Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.656Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.907Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.979Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.024Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.059Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.104Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.175Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.245Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-15_07_37_29-7011471637501784593 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:59.720Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:38:05.392Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:39:38.341Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:39:38.383Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:47:02.391Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:47:02.481Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:47:02.532Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:47:02.595Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:02.932Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:03.010Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:03.087Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:03.159Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:03.193Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:57.397Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:57.441Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:57.472Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-15_07_37_29-7011471637501784593 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 71154216039e4a1d8ccda4f841759318 and timestamp: 1597503011.5470157:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 367
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969539797436
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969539679924
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 52s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/qoktkzn32do5a
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #433
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/433/display/redirect?page=changes>
Changes:
[ningk] [BEAM-10545] HtmlView module
[tysonjh] Update nexmark dashboard links.
[kcweaver] [BEAM-10646] Remove SparkPortableExecutionTest.testExecution.
[kcweaver] [BEAM-10646] Don't wait for test to time out if pipeline fails.
[filiperegadas] [BEAM-10648] Remove unused BigQuery queryTempDataset value
[ningk] Use primitive string[] to replace Array<string> type
[srohde] Fix bug that evicting computed PCollections was changing list while
[noreply] [BEAM-9680] Add Filter with ParDo lesson to Go SDK Katas (#12506)
[noreply] Import WordExtractingDoFn from wordcount_with_metrics
[noreply] [BEAM-9547] Lift associative aggregations. (#12469)
[noreply] Merge pull request #12427 from [BEAM-2855] nexmark python suite
[noreply] [BEAM-10500] Make KeyedTimerDataCoder encode output timestamp (#12535)
[noreply] Extending ApproximateQuantiles functionality to deal with non-uniform
[noreply] fix logic issue in metric name namespace filtering (#12570)
[Valentyn Tymofieiev] Fix Py3 incompatibility in stager.py.
[noreply] Better error on BQ schema parse (#12549)
[ekirpichov] [BEAM-10702] Do not implicitly decompress artifacts
[ekirpichov] Adds a Julia Set test on portable local runner
[ekirpichov] Address review comments
------------------------------------------
[...truncated 26.90 KB...]
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Requirement already satisfied: pyparsing>=2.1.4 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pydot<2,>=1.2.0->apache-beam==2.25.0.dev0) (2.4.7)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting botocore<1.18.0,>=1.17.42
Using cached botocore-1.17.42-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: packaging in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (20.4)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2171959 sha256=767be282752d89cba68f9c56b9c47e817c065814b7a7ec16f26cf56428a52a52
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, certifi, urllib3, chardet, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, more-itertools, wcwidth, atomicwrites, attrs, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, blindspin, colorama, crayons, deprecation, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.42 botocore-1.17.42 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0814125434.1597415846.331285/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0814125434.1597415846.331285/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0814125434.1597415846.331285/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0814125434.1597415846.331285/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-14T14:37:27.821237Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-14_07_37_26-15928784627311346509'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0814125434'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-14T14:37:27.821237Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-14_07_37_26-15928784627311346509]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-14_07_37_26-15928784627311346509
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-14_07_37_26-15928784627311346509?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-14_07_37_26-15928784627311346509 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.067Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.631Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.658Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.688Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.895Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.086Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.130Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.158Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.193Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.226Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.259Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.295Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.320Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.357Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.393Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.431Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.463Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.491Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.521Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.813Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.881Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.942Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.966Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.999Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:32.056Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:32.115Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-14_07_37_26-15928784627311346509 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:53.243Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:57.300Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:39:33.907Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:39:33.952Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:47:00.763Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:47:00.828Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:47:00.881Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:47:00.959Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:00.769Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:00.889Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:00.994Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:01.064Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:01.100Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:51.294Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:51.328Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:51.359Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-14_07_37_26-15928784627311346509 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: eb4783778eb447db8ad9c86fa2200d34 and timestamp: 1597416608.6191347:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 370
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969453395252
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969453278488
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 51s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/2jf7tlgdrbcck
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #432
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/432/display/redirect?page=changes>
Changes:
[srohde] Add max count to utils.to_element_list
[kcweaver] [BEAM-9558] Remove usage of empty data/timers to signify last.
[Maximilian Michels] [BEAM-10676] Use the fire timestamp as the output timestamp for timers
[kcweaver] [BEAM-10681] Set metrics supported in Spark portable runner.
[Maximilian Michels] Use input timestamp as the output timestamp for processing timers
[Maximilian Michels] Add test
[Maximilian Michels] lint
[tobiasz.kedzierski] [BEAM-10686] Simplify names for GCP variables check in GA
[noreply] Fix format string in PipelineValidator (#12522)
[noreply] Fix some typos (#12539)
[noreply] Linkage Checker 1.5.0 (#12545)
[noreply] [BEAM-10684] Fix jdbc cross-language transform (#12543)
[Robert Burke] [BEAM-10610] Clean logging cruft from loop back.
[noreply] Follow the same way that BigQuery handles unspecified or duplicate
[noreply] Merge pull request #12489 from [BEAM-6064] Add an option to avoid
[srohde] Fix BCJ to stop caching when the cache signature has changed.
[daniel.o.programmer] [BEAM-10289] Adding required transform ID to Go channel split.
[noreply] [BEAM-2762] Generate Python coverage reports during pre-commit (#12257)
[daniel.o.programmer] Moving to 2.25.0-SNAPSHOT on master branch.
[noreply] [BEAM-10694] Work around serialization issue with ReaderContext by
------------------------------------------
[...truncated 26.97 KB...]
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.4.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.41
Using cached botocore-1.17.41-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2159547 sha256=3f57a8d111495c0b6bf5b71ba221019c8d5a73e8f0e802bcd014461d718d8e0a
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, idna, certifi, chardet, urllib3, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, packaging, more-itertools, attrs, wcwidth, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, colorama, crayons, blindspin, websocket-client, docker, wrapt, deprecation, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.41 botocore-1.17.41 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0813125359.1597329438.154260/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0813125359.1597329438.154260/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0813125359.1597329438.154260/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0813125359.1597329438.154260/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-13T14:37:20.363720Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-13_07_37_18-5023838095576000549'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0813125359'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-13T14:37:20.363720Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-13_07_37_18-5023838095576000549]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-13_07_37_18-5023838095576000549
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-13_07_37_18-5023838095576000549?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-13_07_37_18-5023838095576000549 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:23.277Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.129Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.169Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.222Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.249Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.389Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.439Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.478Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.515Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.553Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.579Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.615Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.651Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.694Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.723Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.760Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.794Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.835Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:24.912Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-13_07_37_18-5023838095576000549 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:25.581Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:25.664Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:25.724Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:25.756Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:25.802Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:25.887Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:25.948Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:52.960Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:52.989Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:58.448Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:37:58.987Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:39:30.926Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:39:30.956Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:46:54.975Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:46:55.094Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:46:55.146Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:46:55.223Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:48:43.312Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:48:43.369Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:48:43.482Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:48:43.535Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:48:43.555Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:49:31.634Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:49:31.685Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-13T14:49:31.709Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-13_07_37_18-5023838095576000549 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 520be7a6b56b418aadeb598ab2eb6456 and timestamp: 1597330186.2180896:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 361
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969366977836
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969366872626
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 29s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/i6qynfesvtbcs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #431
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/431/display/redirect?page=changes>
Changes:
[michal.walenia] [BEAM-10492] Add missing sideinput handling to DLP transforms
[Maximilian Michels] Revert "Merge pull request #12408: [BEAM-10602] Display Python streaming
[Robin Qiu] Upgrade to ZetaSQL 2020.08.1
[Damian Gadomski] Extending archiveJunit post-commit task with stability history
[Damian Gadomski] Moving /tmp directory cleanup of CI workers to Inventory Jenkins job
[Boyuan Zhang] Workaround of AutoValueSchema doesn't work with SchemaFieldName
[noreply] [BEAM-10572] Eliminate nullability errors from
[noreply] [BEAM-7996] Add Python SqlTransform test that includes a MAP input and
[noreply] [BEAM-10663] Disable python kafka integration tests (#12526)
[Robin Qiu] Change SqlAnalyzer code to use the updated ZetaSQL API
[noreply] Merge pull request #12485 from [BEAM-6064] Improvements to BQ streaming
[noreply] [BEAM-8125] Add verifyDeterministic test to SchemaCoderTest (#12521)
[noreply] [BEAM-10571] Use schemas in ExternalConfigurationPayload (#12481)
[noreply] [BEAM-10644] Mark Beam 2.24.0 as the last release with Py2 and Py35
[noreply] Use new ZetaSQL value create API (#12536)
[noreply] [BEAM-7705] Add BigQuery Java samples (#12118)
[Maximilian Michels] [BEAM-10602] Add latency/checkpoint duration as separate panels in ParDo
------------------------------------------
[...truncated 26.75 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.3.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting botocore<1.18.0,>=1.17.40
Using cached botocore-1.17.40-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2158079 sha256=4d5a5034193eaa8692769505a312ebce25013a116571e74b33f7c0788fb6336c
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, urllib3, certifi, chardet, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, attrs, more-itertools, wcwidth, atomicwrites, packaging, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, websocket-client, docker, colorama, crayons, blindspin, deprecation, wrapt, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.40 botocore-1.17.40 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0812125359.1597243045.531184/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0812125359.1597243045.531184/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0812125359.1597243045.531184/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0812125359.1597243045.531184/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-12T14:37:27.745286Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-12_07_37_26-2766776232286506538'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0812125359'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-12T14:37:27.745286Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-12_07_37_26-2766776232286506538]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-12_07_37_26-2766776232286506538
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-12_07_37_26-2766776232286506538?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-12_07_37_26-2766776232286506538 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:33.682Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.431Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.467Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.515Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.541Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.597Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.692Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.723Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.761Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.784Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.821Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.858Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.892Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.925Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.962Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:34.996Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.026Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.052Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.076Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.412Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.479Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.529Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.568Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.617Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.691Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:35.773Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-12_07_37_26-2766776232286506538 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:37:47.386Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:38:01.084Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:39:45.450Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:39:45.490Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:47:15.416Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:47:15.514Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:47:15.566Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:47:15.632Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:49:13.881Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:49:14.137Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:49:14.375Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:49:14.435Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:49:14.465Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:50:10.448Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:50:10.493Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-12T14:50:10.528Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-12_07_37_26-2766776232286506538 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: db30dbdbe7f64f438dfbed090d325649 and timestamp: 1597243823.5559282:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 373
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969280608204
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969280493292
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 7s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/z5g2lg2yghjhq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #430
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/430/display/redirect?page=changes>
Changes:
[Robin Qiu] Move value conversion logic out of ExpressionConverter
[Robin Qiu] Simplify ZetaSqlBeamTranslationUtils
[Etienne Chauchot] [BEAM-10471] change the test condition for testEstimatedSizeBytes to
[Colm O hEigeartaigh] BEAM-10668 - Replace toLowerCase().equals() with equalsIgnoreCase
[noreply] [BEAM-10361] upgrade Kotlin version in example (#12497)
[noreply] [BEAM-9891] Added ZetaSQL planner support and uploaded 100G data
[noreply] [BEAM-10300] Improve JdbcIOTest.testFluentBackOffConfiguration stability
[noreply] [BEAM-601] Run KinesisIOIT with localstack (#12422)
------------------------------------------
[...truncated 27.00 KB...]
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.3.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting botocore<1.18.0,>=1.17.39
Using cached botocore-1.17.39-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2158044 sha256=e06dc02ad5256bbd520bc34142060f1b18d59b960b23f7ff169c68739f20f6c5
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, chardet, urllib3, idna, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, more-itertools, packaging, atomicwrites, attrs, wcwidth, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, deprecation, wrapt, colorama, crayons, blindspin, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.39 botocore-1.17.39 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0811130639.1597156644.611961/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0811130639.1597156644.611961/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0811130639.1597156644.611961/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0811130639.1597156644.611961/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-11T14:37:26.286086Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-11_07_37_25-1195745182006497943'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0811130639'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-11T14:37:26.286086Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-11_07_37_25-1195745182006497943]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-11_07_37_25-1195745182006497943
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-11_07_37_25-1195745182006497943?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-11_07_37_25-1195745182006497943 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:28.715Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.501Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.528Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.560Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.598Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.675Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.712Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.733Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.765Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.802Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.842Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.872Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.909Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.938Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:29.973Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.008Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.042Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.071Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.097Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.518Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.575Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.609Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.636Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.676Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.734Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:30.792Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-11_07_37_25-1195745182006497943 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:43.461Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:57.815Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 3 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:37:57.865Z: JOB_MESSAGE_DETAILED: Resized **** pool to 3, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:38:03.245Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:38:03.273Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:38:41.004Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:39:39.599Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:39:39.631Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:47:04.090Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:47:04.186Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:47:04.271Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:47:04.335Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:48:57.715Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:48:57.815Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:48:57.934Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:48:57.996Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:48:58.029Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:49:55.545Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:49:55.581Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-11T14:49:55.648Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-11_07_37_25-1195745182006497943 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: b5033743bd2145158bbb17b50335a6c3 and timestamp: 1597157407.0646658:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 370
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969194192222
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969194081756
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 50s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/qlfrlz3potfdm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #429
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/429/display/redirect>
Changes:
------------------------------------------
[...truncated 27.22 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.3.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting botocore<1.18.0,>=1.17.38
Using cached botocore-1.17.38-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2158044 sha256=0abdee886ae1e32ff4cd5a6fcc77f07cdca596731247cd623064d72da30676e4
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, urllib3, certifi, idna, chardet, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, packaging, attrs, more-itertools, atomicwrites, wcwidth, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, websocket-client, docker, deprecation, wrapt, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.38 botocore-1.17.38 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0810125352.1597070244.729032/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0810125352.1597070244.729032/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0810125352.1597070244.729032/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0810125352.1597070244.729032/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-10T14:37:26.255123Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-10_07_37_25-10996910604797877992'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0810125352'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-10T14:37:26.255123Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-10_07_37_25-10996910604797877992]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-10_07_37_25-10996910604797877992
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-10_07_37_25-10996910604797877992?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-10_07_37_25-10996910604797877992 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:28.700Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:29.684Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:29.720Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:29.757Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:29.785Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:29.853Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:29.894Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:29.931Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:29.965Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.002Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.027Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.050Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.073Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.103Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.130Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.161Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.187Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.217Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.242Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.554Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.625Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.660Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.692Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.737Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.804Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:30.873Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-10_07_37_25-10996910604797877992 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:53.505Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:37:58.692Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:39:41.274Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:39:41.299Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:47:00.460Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:47:00.559Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:47:00.627Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:47:00.704Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:48:57.948Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:48:58.060Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:48:58.157Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:48:58.209Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:48:58.232Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:49:44.747Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:49:44.794Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-10T14:49:44.820Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-10_07_37_25-10996910604797877992 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: b60fdabb3dec4277a0e5331b04bc0608 and timestamp: 1597070997.0403585:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 367
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969107792676
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969107678052
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 40s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/kf5x4bl44jnia
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #428
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/428/display/redirect?page=changes>
Changes:
[ettarapp] removed duplicate test
[ettarapp] fixed typo in comments
[ettarapp] removed duplicate tests
[relax] OrderedListState API
------------------------------------------
[...truncated 27.11 KB...]
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.38
Using cached botocore-1.17.38-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2158044 sha256=f7909428d65be387081c1eca2433322fc5d9cfbd13ca9272ff6bc00a69e4fa71
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, chardet, idna, certifi, urllib3, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, packaging, attrs, atomicwrites, more-itertools, wcwidth, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, deprecation, wrapt, colorama, crayons, blindspin, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.38 botocore-1.17.38 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0809125357.1596983840.019230/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0809125357.1596983840.019230/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0809125357.1596983840.019230/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0809125357.1596983840.019230/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-09T14:37:21.811397Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-09_07_37_20-12261486366845572764'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0809125357'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-09T14:37:21.811397Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-09_07_37_20-12261486366845572764]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-09_07_37_20-12261486366845572764
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-09_07_37_20-12261486366845572764?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-09_07_37_20-12261486366845572764 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:24.452Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.298Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.330Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.367Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.402Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.470Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.565Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.592Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.615Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.639Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.662Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.696Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.722Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.758Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.791Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.826Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.862Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.896Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:25.931Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:26.049Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:26.116Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:26.158Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:26.192Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:26.236Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:26.305Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:26.355Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-09_07_37_20-12261486366845572764 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:49.034Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:37:49.882Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:39:34.182Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:39:34.218Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:47:19.741Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:47:19.819Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:47:19.870Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:47:19.921Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:49:13.995Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:49:14.076Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:49:14.192Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:49:14.252Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:49:14.277Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:50:14.726Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:50:14.775Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-09T14:50:14.830Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-09_07_37_20-12261486366845572764 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 75ef6a8f03c34f1096fd97b29d82acd4 and timestamp: 1596984627.657379:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 369
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 11s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/qdud3g3oelfru
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #427
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/427/display/redirect?page=changes>
Changes:
[kevinsijo] documentation(xlang-java): updating docs as per comments
[kevinsijo] documentation(xlang-python): updating docs as per comments
[kevinsijo] documentation(xlang-python): resolving ascii errors
[srohde] Add ElementLimiters to all Cache Managers.
[kamil.galuszka] [BEAM-10647] Fixes get_query_location bug in BigQueryWrapper
[noreply] [BEAM-7390] Add top code snippets (#12482)
[noreply] GH-Actions workflow checks are GCP variables set [depends on BEAM-10599]
[tobiasz.kedzierski] [BEAM-10662] Fix GCP variable check in build python wheels workflow
[noreply] fix precommit errors (#12500)
[Robin Qiu] Support NULL query parameters in ZetaSQL and fix nullable ARRAY bug
[noreply] Merge pull request #12473 from [BEAM-10601] DICOM API Beam IO connector
[kcweaver] [BEAM-10653] Modularize BeamSqlDslUdfUdafTest.
[noreply] [BEAM-10619] Report ratio of implemented pandas tests (#12440)
[daniel.o.programmer] [BEAM-10289] Fixing bug in Go harness split response.
[Boyuan Zhang] [BEAM-9977] Implement ReadFromKafkaViaSDF
[noreply] [BEAM-8460] Exclude category containing failing tests for spark/flink to
[noreply] [BEAM-10656] Enable bundle finalization within the Java direct runner.
------------------------------------------
[...truncated 27.38 KB...]
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting botocore<1.18.0,>=1.17.38
Using cached botocore-1.17.38.tar.gz (6.7 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam, botocore
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2157857 sha256=dce4f6c8812fbb653f28e1579d1ce8439863ef21ef02c80ea5a2298c4660c8a2
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Building wheel for botocore (setup.py): started
Building wheel for botocore (setup.py): finished with status 'done'
Created wheel for botocore: filename=botocore-1.17.38-py2.py3-none-any.whl size=6476992 sha256=ae5aaebb5aa98d9ee3efdb08d1802aa799bfb1912ce537705953f5e1f4cb5117
Stored in directory: /home/jenkins/.cache/pip/wheels/4c/2e/c2/67d474b590ecb1920117456c1084dca3cabf7cf3634afa44a7
Successfully built apache-beam botocore
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, chardet, certifi, idna, urllib3, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, more-itertools, packaging, wcwidth, attrs, atomicwrites, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, colorama, crayons, blindspin, deprecation, wrapt, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.38 botocore-1.17.38 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0808125353.1596897446.238658/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0808125353.1596897446.238658/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0808125353.1596897446.238658/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0808125353.1596897446.238658/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-08T14:37:29.279125Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-08_07_37_26-5292319146438785977'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0808125353'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-08T14:37:29.279125Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-08_07_37_26-5292319146438785977]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-08_07_37_26-5292319146438785977
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-08_07_37_26-5292319146438785977?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-08_07_37_26-5292319146438785977 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.029Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.731Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.757Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.787Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.812Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.866Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.902Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.924Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.950Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.973Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:32.996Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.017Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.044Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.065Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.089Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.112Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.132Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.159Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.178Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.277Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.326Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.359Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.384Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.416Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.462Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:37:33.518Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-08_07_37_26-5292319146438785977 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:38:01.286Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:38:02.520Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:39:31.797Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:39:31.828Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:47:06.840Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:47:06.928Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:47:06.979Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:47:07.051Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:48:58.647Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:48:58.722Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:48:58.844Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:48:58.902Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:48:58.938Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:49:47.660Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:49:47.709Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-08T14:49:47.739Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-08_07_37_26-5292319146438785977 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 8782c2ae06c342e3a271f5a13d45a966 and timestamp: 1596898195.0570006:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 374
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224968934993204
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224968934884496
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 40s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/gvyaumv4ms2hm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #426
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/426/display/redirect?page=changes>
Changes:
[piotr.szuberski] [BEAM-10135][BEAM-10136] Refactor jdbc external transform registrar
[douglas.damon] Add Additional Parameters lesson to Go SDK Katas
[michal.walenia] [BEAM-9421] Add Java snippets to NLP documentation.
[douglas.damon] Edit lesson name and task description
[Rui Wang] [BEAM-10633] UdfImpl should be able to return java.util.List.
[piotr.szuberski] [BEAM-10543] Add new parameters to Kafka Read cross language
[piotr.szuberski] [BEAM-10543] Add new parameters to python wrapper of Kafka Read
[piotr.szuberski] [BEAM-10543] Modify Kafka python cross-language integration test to use
[piotr.szuberski] [BEAM-10543] Run Kafka cross-language integration test in python
[Robert Burke] [BEAM-9615] Add initial Schema to Go conversions.
[dcavazos] [BEAM-7390] Add sum code snippets
[douglas.damon] Additional edits to task description
[noreply] [BEAM-10258] Support type hint annotations on PTransform's expand()
[noreply] [BEAM-10522] Added SnowflakeIO connector guide (#12296)
[douglas.damon] Update stepik course
[noreply] [BEAM-10240] Support ZetaSQL DATETIME functions in BeamSQL (#12348)
[Robert Burke] [BEAM-9615] Improve error handling on schema conv
[noreply] Updating changes.md (#12424)
[noreply] [BEAM-10645] Create context for allowing non-parallel dataframe
[noreply] [BEAM-10630] Check load tests as part of the release process (#12455)
------------------------------------------
[...truncated 27.08 KB...]
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting botocore<1.18.0,>=1.17.37
Using cached botocore-1.17.37-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2151686 sha256=eb8430914b0bf9b0c7256ba4d5d400e394265f1261f0f6794b60cf48ff1521cd
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, certifi, chardet, urllib3, idna, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, packaging, attrs, atomicwrites, more-itertools, wcwidth, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, colorama, crayons, websocket-client, docker, deprecation, blindspin, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.37 botocore-1.17.37 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0807125414.1596811051.750151/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0807125414.1596811051.750151/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0807125414.1596811051.750151/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0807125414.1596811051.750151/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-07T14:37:33.517126Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-07_07_37_32-9066875279145406151'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0807125414'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-07T14:37:33.517126Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-07_07_37_32-9066875279145406151]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-07_07_37_32-9066875279145406151
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-07_07_37_32-9066875279145406151?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-07_07_37_32-9066875279145406151 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:36.154Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:36.938Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.013Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.049Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.077Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.149Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.180Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.207Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.245Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.281Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.318Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.359Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.395Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.422Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.450Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.483Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.511Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.535Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.565Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.874Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.928Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:37.977Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:38.002Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:38.047Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:38.102Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:38.162Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-07_07_37_32-9066875279145406151 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:37:53.259Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:38:04.167Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:39:46.689Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:39:46.712Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:47:00.524Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:47:00.609Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:47:00.670Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:47:00.734Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:48:59.986Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:49:00.063Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:49:00.184Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:49:00.243Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:49:00.275Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:50:02.319Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:50:02.386Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-07T14:50:02.422Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-07_07_37_32-9066875279145406151 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: f6611ec4197340c482d0170d47737d80 and timestamp: 1596811814.3441155:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 361
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 57s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/hlrjvrl2226li
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #425
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/425/display/redirect?page=changes>
Changes:
[tysonjh] Update google-api-services* versions and google-client version to
[tysonjh] Set orderingKey field by method, not field name.
[Robert Burke] [BEAM-7996] Add map & nil encoding to Go SDK.
[Robert Bradshaw] Simplify common patterns for pandas methods.
[Robert Bradshaw] Use new infrastructure to simplify pandas implementation.
[noreply] [BEAM-7996] Add support for MapType and Nulls in container types for
------------------------------------------
[...truncated 27.19 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting botocore<1.18.0,>=1.17.36
Using cached botocore-1.17.36-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.0-py2.py3-none-any.whl (42 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2148673 sha256=0f36b2345b7bdf9fd2722a1ff557bcab24463549cbcf15062b088d45eb1400fc
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, urllib3, certifi, idna, chardet, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, more-itertools, wcwidth, attrs, packaging, atomicwrites, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, websocket-client, docker, blindspin, deprecation, wrapt, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.36 botocore-1.17.36 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.0 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.0 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0806125357.1596726626.733281/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0806125357.1596726626.733281/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0806125357.1596726626.733281/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0806125357.1596726626.733281/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-06T15:10:28.788081Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-06_08_10_27-5531161875166985047'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0806125357'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-06T15:10:28.788081Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-06_08_10_27-5531161875166985047]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-06_08_10_27-5531161875166985047
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-06_08_10_27-5531161875166985047?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-06_08_10_27-5531161875166985047 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:31.611Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:32.552Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:32.584Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:32.831Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:32.899Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.009Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.071Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.108Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.146Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.187Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.222Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.257Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.281Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.307Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.346Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.382Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.415Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.448Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.470Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.615Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.699Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.744Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.810Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.862Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.934Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:10:33.995Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-06_08_10_27-5531161875166985047 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:11:01.472Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 3 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:11:01.505Z: JOB_MESSAGE_DETAILED: Resized **** pool to 3, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:11:05.013Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:11:06.910Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:12:46.136Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:12:46.176Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:21:35.469Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:21:35.597Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:21:35.644Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:21:35.703Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:23:40.922Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:23:40.987Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:23:41.132Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:23:41.197Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:23:41.227Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:24:33.366Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:24:33.416Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-06T15:24:33.456Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-06_08_10_27-5531161875166985047 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 9cf4ff28b5cc48e58e14f91ebaac5896 and timestamp: 1596727484.7137659:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 446
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 15m 19s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/una7up2ivuttm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #424
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/424/display/redirect?page=changes>
Changes:
[jx643] fixing the IOException handling
[jiadaixia] change getErrorInfo into seperate method
[jiadaixia] delete unused import
[jiadaixia] delete empty line in import
[jiadaixia] spotless change
[jiadaixia] naming change
[jiadaixia] change to vendored gauva class
[jiadaixia] add unit test
[jiadaixia] remove redundent code
[jiadaixia] further simplifies getErrorInfo
[jiadaixia] spotless fix
[jiadaixia] update
[jiadaixia] spotless
[jiadaixia] add comment
[Robert Bradshaw] Move wordcount with metrics to its own file.
[Robert Bradshaw] Cleanup WordCount example.
[jiadaixia] change test names
[jiadaixia] spotless
[jiadaixia] style change
[jiadaixia] add failure
[srohde] Add ElementLimiters which allows the cache to prematurely based on read
[jiadaixia] change exception handling
[jiadaixia] add jira issue
[srohde] Remove start_secs parameter.
[daniel.o.programmer] [BEAM-10289] Go Dynamic splitting full implementation.
[daniel.o.programmer] [BEAM-10289] Avoiding blocking when dynamic splitting in Go.
[dcavazos] [BEAM-7390] Add sample code snippets
[ihr] Fix add field method in SQL walkthrough
[Andrew Pilloud] [BEAM-10470] Handle null state from waitUntilFinish
[noreply] Merge pull request #12203 from [BEAM-6928] Make Python SDK custom Sink
[ningk] [BEAM-10635] Fix forward the google-api-core version
[noreply] [BEAM-10618] subprocess_server.py: Fallback to AF_INET6 family when
[noreply] [BEAM-10629] Added KnownBuilderInstances to ExternalTransformRegistrar
[noreply] [BEAM-10637] fix: test stream service start/stop (#12464)
[noreply] Migrate shared tag from tfx-bsl (#12468)
[noreply] Merge pull request #12149: [BEAM-9897] Add cross-language support to
[noreply] Merge pull request #12151: [BEAM-9896] Add streaming for
------------------------------------------
[...truncated 27.21 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.35
Using cached botocore-1.17.35-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.0-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2146671 sha256=064a13038d98826178e63ca65352d1d0cced6380388af9412025bfb8803266dd
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, certifi, idna, urllib3, chardet, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, more-itertools, attrs, wcwidth, packaging, atomicwrites, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, deprecation, blindspin, wrapt, colorama, crayons, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.35 botocore-1.17.35 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.0 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.0 google-cloud-datastore-1.13.2 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.0 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0805142445.1596638241.811522/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0805142445.1596638241.811522/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0805142445.1596638241.811522/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0805142445.1596638241.811522/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-05T14:37:23.751784Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-05_07_37_22-16285829067567199153'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0805142445'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-05T14:37:23.751784Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-05_07_37_22-16285829067567199153]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-05_07_37_22-16285829067567199153
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-05_07_37_22-16285829067567199153?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-05_07_37_22-16285829067567199153 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:27.137Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:27.884Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.007Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.043Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.073Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.136Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.176Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.212Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.244Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.277Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.310Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.345Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.380Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.416Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.450Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.486Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.531Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.567Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.604Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.728Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.782Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.819Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.853Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.893Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:28.963Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:29.042Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-05_07_37_22-16285829067567199153 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:54.854Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:37:56.348Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:39:41.630Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:39:41.675Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:47:38.181Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:47:38.277Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:47:38.339Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:47:38.414Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:49:40.698Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:49:40.767Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:49:40.887Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:49:40.940Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:49:40.978Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:50:42.549Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:50:42.600Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-05T14:50:42.637Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-05_07_37_22-16285829067567199153 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 8a1e91821aa24678b820dd0108dd67c4 and timestamp: 1596639054.8336852:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 386
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224968675835208
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224968675715864
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 37s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/ao6qnfj73dzhe
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #423
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/423/display/redirect?page=changes>
Changes:
[Boyuan Zhang] Refactor SplittableParDoExpander.
[kcweaver] [BEAM-7632] Add Python quickstart instructions for Flink and Spark.
[ningk] Fixed pcoll visualization in datatable
[hsuryawirawan] Update Python Katas on Stepik based on the latest pipeline "with" style
[Maximilian Michels] [BEAM-10622] Prefix Gradle paths with a colon for user-facing output
[qz689] [BEAM-9543] built the basis for Match_Recog
[qz689] [BEAM-9543] built the basis for Match_Recog
[qz689] [BEAM-9543] implemented `partition by`
[qz689] [BEAM-9543] implemented `order by`
[qz689] [BEAM-9543] fixed `order by` coder issue
[qz689] [BEAM-9543] fixed `order by` coder issue
[qz689] [BEAM-9543] applied regex pattern match
[qz689] [BEAM-9543] applied regex pattern match
[qz689] [BEAM-9543] fixed sortKey serialization problem
[qz689] [BEAM-9543] fixed sortKey serialization problem
[qz689] [BEAM-9543] fixed serialization problem
[qz689] [BEAM-9543] recognized simple pattern
[qz689] [BEAM-9543] recognized simple pattern
[qz689] [BEAM-9543] fixed code style
[qz689] [BEAM-9543] supported regex quantifier
[qz689] [BEAM-9543] added javadoc
[qz689] [BEAM-9543] removed CEPTypeName.java
[qz689] [BEAM-9543] added Measures implementation (unfinished)
[qz689] [BEAM-9543] added Measures implementation
[qz689] [BEAM-9543] added Measures implementation
[qz689] [BEAM-9543] fixed minor issues
[qz689] [BEAM-9543] fixed minor style issues
[noreply] [BEAM-10599] Add documentation about CI on GitHub Action (#12405)
[noreply] Fix link for S3FileSystem (#12450)
[dcavazos] [BEAM-7390] Add min code snippets
[noreply] [BEAM-7390] Add max code snippets (#12409)
[noreply] [BEAM-7390] Add mean code snippets (#12437)
[borzoo.esmailloo] [BEAM-10499] Adds a descriptive toString to SamzaRunner KeyedTimerData
[noreply] [BEAM-9839] OnTimerContext should not create a new one when processing
[noreply] Fix dictionary changes size error in pickler.py (#12458)
[noreply] [BEAM-9891] TPC-DS module initialization, tables and queries stored
[Boyuan Zhang] Address comments.
[Boyuan Zhang] Fix SDF/Process input id.
[noreply] Interactive: clean up when pipeline is out of scope (#12339)
[noreply] Merge pull request #12331 [BEAM-10601] DICOM API Beam IO connector
[noreply] [BEAM-10631] Fix performance of Schema#indexOf (#12456)
------------------------------------------
[...truncated 27.33 KB...]
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.34
Using cached botocore-1.17.34-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.0-py2.py3-none-any.whl (42 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2139384 sha256=1ed473b97eebeb5c8100bd5bab6ad691c48ae29b8d8220d3f4ac3ea3d186df6d
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, certifi, urllib3, chardet, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, packaging, more-itertools, atomicwrites, attrs, wcwidth, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, deprecation, websocket-client, docker, colorama, crayons, wrapt, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.34 botocore-1.17.34 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.0 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.3.0 google-cloud-datastore-1.13.2 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.0 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0804125351.1596551842.106123/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0804125351.1596551842.106123/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0804125351.1596551842.106123/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0804125351.1596551842.106123/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-04T14:37:23.900877Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-04_07_37_22-15598061054323363970'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0804125351'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-04T14:37:23.900877Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-04_07_37_22-15598061054323363970]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-04_07_37_22-15598061054323363970
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-04_07_37_22-15598061054323363970?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-04_07_37_22-15598061054323363970 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:26.026Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:26.794Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:26.825Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:26.857Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:26.881Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:26.926Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:26.964Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:26.987Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.011Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.049Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.076Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.109Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.137Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.169Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.198Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.222Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.244Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.268Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.295Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.461Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.513Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.560Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.579Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.616Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.660Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:27.711Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-04_07_37_22-15598061054323363970 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:55.606Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:37:55.632Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:38:01.015Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:38:04.089Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:39:30.793Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:39:30.825Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:46:56.791Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:46:56.930Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:46:56.991Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:46:57.056Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:49:01.693Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:49:01.746Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:49:01.856Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:49:01.915Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:49:01.944Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:50:00.989Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:50:01.025Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-04T14:50:01.056Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-04_07_37_22-15598061054323363970 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: ef18a71dbb9e4dc98a2bfe399e7eee4d and timestamp: 1596552609.7521846:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 383
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224968589396298
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224968589274502
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 52s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/6v3ijxjh6xoiu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #422
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/422/display/redirect?page=changes>
Changes:
[Etienne Chauchot] [BEAM-10471] issue a JMX call to cassandra
[Etienne Chauchot] [BEAM-10471] fix a wrong comment
[Maximilian Michels] [BEAM-10602] Use python_streaming_pardo_5 table for latency results
------------------------------------------
[...truncated 27.32 KB...]
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.33
Using cached botocore-1.17.33-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.0-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-auth<2.0dev,>=1.19.1
Using cached google_auth-1.20.0-py2.py3-none-any.whl (91 kB)
Collecting google-crc32c>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2127755 sha256=2645044332f11d715b0f8dba032b5d3b9e2e4fbf4ae0c89ee803d5f5f5a5e0fe
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, chardet, idna, certifi, urllib3, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, googleapis-common-protos, google-auth, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, more-itertools, wcwidth, packaging, atomicwrites, attrs, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, deprecation, blindspin, websocket-client, docker, colorama, crayons, wrapt, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.33 botocore-1.17.33 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.0 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.3.0 google-cloud-datastore-1.13.2 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.0 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0803140036.1596465436.628445/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0803140036.1596465436.628445/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0803140036.1596465436.628445/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0803140036.1596465436.628445/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-03T14:37:18.414875Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-03_07_37_17-18432873861416994835'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0803140036'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-03T14:37:18.414875Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-03_07_37_17-18432873861416994835]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-03_07_37_17-18432873861416994835
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-03_07_37_17-18432873861416994835?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-03_07_37_17-18432873861416994835 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:24.438Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.134Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.229Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.269Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.312Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.380Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.425Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.448Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.482Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.511Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.567Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.601Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.639Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.673Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.705Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.736Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.819Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.854Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:25.881Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:26.020Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:26.093Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:26.137Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:26.161Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:26.206Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:26.269Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:26.352Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-03_07_37_17-18432873861416994835 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:52.282Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:52.320Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:37:57.706Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:38:01.315Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:39:27.219Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:39:27.254Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:47:10.361Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:47:10.445Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:47:10.490Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:47:10.556Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:49:16.767Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:49:16.844Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:49:16.967Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:49:17.033Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:49:17.066Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:50:11.232Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:50:11.318Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-03T14:50:11.402Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-03_07_37_17-18432873861416994835 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 5526c4b035cd477dbb96662409ef7287 and timestamp: 1596466219.217844:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 401
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224968503011368
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224968502887812
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 2s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/s56h6iq6ps3oq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #421
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/421/display/redirect>
Changes:
------------------------------------------
[...truncated 27.28 KB...]
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.1)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting botocore<1.18.0,>=1.17.33
Using cached botocore-1.17.33-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.0-py2.py3-none-any.whl (42 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting google-auth<2.0dev,>=1.19.1
Using cached google_auth-1.20.0-py2.py3-none-any.whl (91 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2127755 sha256=081f018be71ec14b39fe4a068aa4c6f44e11fad0a6b25e6c9d76296b73148c9c
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, urllib3, chardet, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, wcwidth, attrs, packaging, atomicwrites, more-itertools, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, blindspin, colorama, crayons, websocket-client, docker, deprecation, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.33 botocore-1.17.33 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.0 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.3.0 google-cloud-datastore-1.13.2 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.0 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0802125411.1596379039.498119/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0802125411.1596379039.498119/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0802125411.1596379039.498119/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0802125411.1596379039.498119/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-02T14:37:21.447340Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-02_07_37_19-16564730461235922718'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0802125411'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-02T14:37:21.447340Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-02_07_37_19-16564730461235922718]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-02_07_37_19-16564730461235922718
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-02_07_37_19-16564730461235922718?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-02_07_37_19-16564730461235922718 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:24.089Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:24.799Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:24.830Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:24.863Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.034Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.102Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.145Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.180Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.216Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.243Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.271Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.310Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.358Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.385Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.411Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.437Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.467Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.495Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.517Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.673Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.740Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.781Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.815Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.858Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.919Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:25.988Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-02_07_37_19-16564730461235922718 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:40.634Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:49.291Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 3 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:49.311Z: JOB_MESSAGE_DETAILED: Resized **** pool to 3, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:37:54.708Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:39:37.461Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:39:37.491Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:47:10.365Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:47:10.424Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:47:10.480Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:47:10.527Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:49:13.014Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:49:13.067Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:49:13.177Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:49:13.235Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:49:13.259Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:50:03.883Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:50:03.938Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-02T14:50:03.964Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-02_07_37_19-16564730461235922718 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: cdaa4777dc7948a7a99bad94b0ff9d0c and timestamp: 1596379817.2521696:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 376
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224968416606688
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224968416487912
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 0s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/ygospxquugssg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #420
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/420/display/redirect?page=changes>
Changes:
[srohde] Gracefully stop the TestStream RPC if the job stops.
[piotr.szuberski] [BEAM-10591] Fix flaky jdbc test by adding retries to container creation
[piotr.szuberski] Add logging after catching an exception
[kcweaver] Update Flink Version Compatibility table
[chamikaramj] Sets the region when looking up a BQ job
[kcweaver] fixup! Update Flink Version Compatibility table
[kcweaver] [BEAM-9199] Update Dataflow --region option description.
[noreply] Enable all Jenkins jobs triggering for committers (#12407)
[ningk] Remove trailing whitespace from README
[noreply] Widen ranges for GCP libraries (#12198)
------------------------------------------
[...truncated 27.34 KB...]
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.24.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.24.0.dev0) (49.2.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.33
Using cached botocore-1.17.33-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.0-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.7.0)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (1.9.0)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (0.13.1)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting docker
Using cached docker-4.2.2-py2.py3-none-any.whl (144 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting google-auth<2.0dev,>=1.19.1
Using cached google_auth-1.20.0-py2.py3-none-any.whl (91 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.24.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.24.0.dev0-py3-none-any.whl size=2127755 sha256=2e934ec8bf0ac2cb9b738dc3ce96d929f6cb8bc6561f884b67174ba4baaf61c1
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, urllib3, certifi, chardet, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, packaging, atomicwrites, wcwidth, more-itertools, attrs, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, websocket-client, docker, blindspin, deprecation, colorama, crayons, wrapt, testcontainers, apache-beam
Successfully installed apache-beam-2.24.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.33 botocore-1.17.33 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.2.2 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.0 google-apitools-0.5.31 google-auth-1.20.0 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.3.0 google-cloud-datastore-1.13.2 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.0 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.24.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0801125358.1596292636.012792/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0801125358.1596292636.012792/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0801125358.1596292636.012792/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0801125358.1596292636.012792/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-01T14:37:17.828656Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-01_07_37_16-6233350765665033242'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0801125358'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-01T14:37:17.828656Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-01_07_37_16-6233350765665033242]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-01_07_37_16-6233350765665033242
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-01_07_37_16-6233350765665033242?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-01_07_37_16-6233350765665033242 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:20.444Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.065Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.142Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.172Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.199Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.294Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.351Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.375Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.399Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.422Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.444Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.469Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.490Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.515Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.538Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.562Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.588Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.607Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.631Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.867Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.925Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.958Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:21.981Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:22.022Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:22.074Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:22.121Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-01_07_37_16-6233350765665033242 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:51.372Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 3 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:51.400Z: JOB_MESSAGE_DETAILED: Resized **** pool to 3, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:56.741Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:56.765Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:37:58.264Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:38:02.083Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:39:28.333Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:39:28.359Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:46:56.748Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:46:56.833Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:46:56.880Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:46:56.956Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:49:03.639Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:49:03.687Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:49:03.779Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:49:03.830Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:49:03.857Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:49:57.427Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:49:57.472Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-01T14:49:57.507Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-01_07_37_16-6233350765665033242 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: bc2971ba57854183bf804eed8443419e and timestamp: 1596293413.6344545:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 381
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224968330196892
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 57s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/xp5zunad7ifuc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org