You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2020/08/14 14:50:10 UTC
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #433
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/433/display/redirect?page=changes>
Changes:
[ningk] [BEAM-10545] HtmlView module
[tysonjh] Update nexmark dashboard links.
[kcweaver] [BEAM-10646] Remove SparkPortableExecutionTest.testExecution.
[kcweaver] [BEAM-10646] Don't wait for test to time out if pipeline fails.
[filiperegadas] [BEAM-10648] Remove unused BigQuery queryTempDataset value
[ningk] Use primitive string[] to replace Array<string> type
[srohde] Fix bug that evicting computed PCollections was changing list while
[noreply] [BEAM-9680] Add Filter with ParDo lesson to Go SDK Katas (#12506)
[noreply] Import WordExtractingDoFn from wordcount_with_metrics
[noreply] [BEAM-9547] Lift associative aggregations. (#12469)
[noreply] Merge pull request #12427 from [BEAM-2855] nexmark python suite
[noreply] [BEAM-10500] Make KeyedTimerDataCoder encode output timestamp (#12535)
[noreply] Extending ApproximateQuantiles functionality to deal with non-uniform
[noreply] fix logic issue in metric name namespace filtering (#12570)
[Valentyn Tymofieiev] Fix Py3 incompatibility in stager.py.
[noreply] Better error on BQ schema parse (#12549)
[ekirpichov] [BEAM-10702] Do not implicitly decompress artifacts
[ekirpichov] Adds a Julia Set test on portable local runner
[ekirpichov] Address review comments
------------------------------------------
[...truncated 26.90 KB...]
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Requirement already satisfied: pyparsing>=2.1.4 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pydot<2,>=1.2.0->apache-beam==2.25.0.dev0) (2.4.7)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting botocore<1.18.0,>=1.17.42
Using cached botocore-1.17.42-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: packaging in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (20.4)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2171959 sha256=767be282752d89cba68f9c56b9c47e817c065814b7a7ec16f26cf56428a52a52
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, certifi, urllib3, chardet, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, more-itertools, wcwidth, atomicwrites, attrs, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, blindspin, colorama, crayons, deprecation, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.42 botocore-1.17.42 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.14.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0814125434.1597415846.331285/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0814125434.1597415846.331285/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0814125434.1597415846.331285/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0814125434.1597415846.331285/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-14T14:37:27.821237Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-14_07_37_26-15928784627311346509'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0814125434'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-14T14:37:27.821237Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-14_07_37_26-15928784627311346509]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-14_07_37_26-15928784627311346509
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-14_07_37_26-15928784627311346509?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-14_07_37_26-15928784627311346509 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.067Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.631Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.658Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.688Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:30.895Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.086Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.130Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.158Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.193Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.226Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.259Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.295Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.320Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.357Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.393Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.431Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.463Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.491Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.521Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.813Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.881Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.942Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.966Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:31.999Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:32.056Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:32.115Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-14_07_37_26-15928784627311346509 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:53.243Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:37:57.300Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:39:33.907Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:39:33.952Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:47:00.763Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:47:00.828Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:47:00.881Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:47:00.959Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:00.769Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:00.889Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:00.994Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:01.064Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:01.100Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:51.294Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:51.328Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-14T14:49:51.359Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-14_07_37_26-15928784627311346509 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: eb4783778eb447db8ad9c86fa2200d34 and timestamp: 1597416608.6191347:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 370
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969453395252
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969453278488
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 51s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/2jf7tlgdrbcck
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal :
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #445
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/445/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #444
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/444/display/redirect?page=changes>
Changes:
[Udi Meiri] Silence red PYTHONPATH warning from tox
[Robin Qiu] Propagate BigQuery streaming insert throttled time to Dataflow worker in
[Kamil Wasilewski] [BEAM-10524] JSON decoder for ReadFromBigQuery supports repeatable
[noreply] Merge pull request #12415 from [BEAM-10603] Add the RecordingManager and
[noreply] [BEAM-10777] Add two blog posts detailing changes to the type hints
------------------------------------------
[...truncated 27.33 KB...]
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting botocore<1.18.0,>=1.17.48
Using cached botocore-1.17.48-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam, boto3
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2204369 sha256=0d05dcaa61c6d4bc8c03cbd666b1c26c677b0aa37dda91363932f08121b8169a
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Building wheel for boto3 (setup.py): started
Building wheel for boto3 (setup.py): finished with status 'done'
Created wheel for boto3: filename=boto3-1.14.48-py2.py3-none-any.whl size=127852 sha256=2163e9b0d03fec61a7fcf4c46fe818a4f7f6e9cc596f8aea33e197fcb64c48ac
Stored in directory: /home/jenkins/.cache/pip/wheels/b0/f8/d9/d2a816eb3b5fed353cf3e7746d2c3bd0cac694045d063a2b53
Successfully built apache-beam boto3
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, certifi, chardet, urllib3, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, more-itertools, attrs, wcwidth, packaging, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, deprecation, websocket-client, docker, colorama, crayons, blindspin, wrapt, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.48 botocore-1.17.48 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.3 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0825125343.1598366237.244709/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0825125343.1598366237.244709/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0825125343.1598366237.244709/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0825125343.1598366237.244709/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-25T14:37:19.005549Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-25_07_37_17-11432471775787433052'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0825125343'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-25T14:37:19.005549Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-25_07_37_17-11432471775787433052]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-25_07_37_17-11432471775787433052
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-25_07_37_17-11432471775787433052?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-25_07_37_17-11432471775787433052 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:21.779Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:22.728Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:22.765Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:22.817Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:22.942Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.025Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.062Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.098Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.135Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.183Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.216Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.353Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.381Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.407Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.433Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.470Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.502Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.533Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.619Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.846Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.917Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.956Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:23.980Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:24.012Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:24.075Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:24.135Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-25_07_37_17-11432471775787433052 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:49.553Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:53.072Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:53.107Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:37:58.418Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:39:31.637Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:39:31.663Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:46:59.006Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:46:59.099Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:46:59.142Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:46:59.196Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.042Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.205Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.284Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.419Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:04.450Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:57.995Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:58.029Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-25T14:49:58.063Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-25_07_37_17-11432471775787433052 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 9eb2c8fb6b5c433886fc2f7c094d8de3 and timestamp: 1598367014.8794644:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 372
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224970403798592
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224970403676592
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 57s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/mnx7ntwhfumsq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #443
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/443/display/redirect?page=changes>
Changes:
[tobiasz.kedzierski] [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac
------------------------------------------
[...truncated 27.03 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting botocore<1.18.0,>=1.17.47
Using cached botocore-1.17.47-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2197167 sha256=76d3fcb367bf0b96464e2287ee013686707bcd6ae1e8d8693e4bba977f76ee8c
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, urllib3, chardet, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, attrs, packaging, atomicwrites, more-itertools, wcwidth, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, deprecation, wrapt, websocket-client, docker, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.47 botocore-1.17.47 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.3 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0824125344.1598279836.241577/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0824125344.1598279836.241577/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0824125344.1598279836.241577/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0824125344.1598279836.241577/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-24T14:37:17.876845Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-24_07_37_16-3221239520918555375'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0824125344'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-24T14:37:17.876845Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-24_07_37_16-3221239520918555375]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-24_07_37_16-3221239520918555375
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-24_07_37_16-3221239520918555375?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-24_07_37_16-3221239520918555375 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:20.751Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.675Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.701Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.759Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.787Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.833Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.874Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.900Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.927Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.947Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.973Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:21.999Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.021Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.049Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.069Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.098Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.122Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.149Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.185Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.355Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.413Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.453Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.473Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.507Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.563Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:22.627Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-24_07_37_16-3221239520918555375 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:37.104Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:37:48.882Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:39:26.056Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:39:26.078Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:46:50.721Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:46:50.825Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:46:50.912Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:46:50.981Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:48.854Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:48.909Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:48.989Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:49.055Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:48:49.082Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:49:48.572Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:49:48.644Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-24T14:49:48.674Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-24_07_37_16-3221239520918555375 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: bfbcd0211acd4f4094c2966889476c50 and timestamp: 1598280603.722677:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 367
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224970317383426
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224970317268208
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 46s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/nz7aqj342ggss
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #442
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/442/display/redirect?page=changes>
Changes:
[noreply] [BEAM-7587] Spark portable streaming (#12157)
[noreply] Merge pull request #12653 from [BEAM-10758] Centralized Glob Translation
------------------------------------------
[...truncated 26.97 KB...]
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.47
Using cached botocore-1.17.47-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2197167 sha256=a887d1b2964f58393c4d797c4a3fff7262d8b8cb113ca677464017d0d29800a7
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, certifi, idna, chardet, urllib3, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, attrs, more-itertools, wcwidth, packaging, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, deprecation, blindspin, colorama, crayons, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.47 botocore-1.17.47 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0823125359.1598193436.224736/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0823125359.1598193436.224736/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0823125359.1598193436.224736/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0823125359.1598193436.224736/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-23T14:37:17.721574Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-23_07_37_16-5148909312103907913'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0823125359'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-23T14:37:17.721574Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-23_07_37_16-5148909312103907913]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-23_07_37_16-5148909312103907913
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-23_07_37_16-5148909312103907913?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-23_07_37_16-5148909312103907913 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:20.227Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.047Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.079Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.114Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.134Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.203Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.234Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.266Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.295Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.329Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.363Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.398Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.430Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.465Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.498Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.532Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.566Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.601Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.635Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.780Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.841Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.885Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.917Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:21.960Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:22.042Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:22.102Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-23_07_37_16-5148909312103907913 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:38.482Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:37:50.738Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:39:34.534Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:39:34.565Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:46:50.635Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:46:50.700Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:46:50.793Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:46:50.858Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.543Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.612Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.672Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.730Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:48:42.767Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:49:41.253Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:49:41.298Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-23T14:49:41.336Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-23_07_37_16-5148909312103907913 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 73d874d3b3c44293af7eb47ac6675ad7 and timestamp: 1598194193.5487041:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 359
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224970230977252
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 36s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/73jgdqpzxlxkm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #441
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/441/display/redirect?page=changes>
Changes:
[Udi Meiri] Color Jenkins logs support
[ningk] Added a whitespace lint precommit job.
[ningk] Removed trailing whitespaces from all markdown and gradle build files.
[ningk] Updated jenkins precommit job README, added comments of what needs to be
[ningk] Removed trailing whitespaces from HEAD changes
[Alan Myrvold] [BEAM-10049] Add licenses for go dependencies in python, java, and go
[Robin Qiu] Fix bug in JoinScanWithRefConverter
[noreply] Propagate BigQuery streaming insert throttled time to Dataflow worker in
[tysonjh] [BEAM-10751] Revert "Extending archiveJunit post-commit task with
[noreply] [BEAM-10764] Make is_in_ipython robust (#12641)
[noreply] [BEAM-10766] Fix flake where ByteString coercion could modify cached
[ningk] [BEAM-10771] Added Whitespacelint job to pull request template
[noreply] Use setCoder, not setSchema (#12662)
[noreply] [BEAM-10703] Prepare Dataflow Java runner for shardable states (#12578)
[noreply] [BEAM-10549] Improve runtime type checking performance for the Python
[Kyle Weaver] [BEAM-10460] Remove SparkPortableExecutionTest altogether.
[noreply] [BEAM-10788] Also disable immutability checking for transforms within
[noreply] [BEAM-5715, BEAM-8862, BEAM-8702] Removes all references to grpc_all and
[noreply] [BEAM-10761] Prefer TestPubsub#assertSubscriptionEventuallyExists when
[noreply] [BEAM-10654] Implemented ExternalSchemaIOTransformRegistrar for jdbc
[noreply] Merge pull request #12580 from [BEAM-2855] nexmark python suite
[noreply] [BEAM-9918] Adding tests and documentation to xlang components (#12667)
------------------------------------------
[...truncated 26.89 KB...]
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.47
Using cached botocore-1.17.47-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2197167 sha256=c779d8989b9142583c7827787c5ca6f585355a78d87479168bea25986b261bc4
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, urllib3, idna, chardet, certifi, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, attrs, wcwidth, atomicwrites, packaging, more-itertools, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, wrapt, websocket-client, docker, deprecation, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.47 botocore-1.17.47 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0822125344.1598107038.368177/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0822125344.1598107038.368177/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0822125344.1598107038.368177/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0822125344.1598107038.368177/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-22T14:37:19.972862Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-22_07_37_18-7377215218196749982'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0822125344'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-22T14:37:19.972862Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-22_07_37_18-7377215218196749982]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-22_07_37_18-7377215218196749982
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-22_07_37_18-7377215218196749982?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-22_07_37_18-7377215218196749982 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:22.496Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.369Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.406Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.443Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.478Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.528Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.562Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.592Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.622Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.651Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.683Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.719Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.750Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.782Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.817Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.844Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.879Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.906Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:23.937Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.070Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.139Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.181Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.207Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.246Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.289Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:24.351Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-22_07_37_18-7377215218196749982 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:37:57.238Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:38:00.145Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:39:36.809Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:39:36.836Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:47:02.455Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:47:02.589Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:47:02.644Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:47:02.716Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.004Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.067Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.140Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.195Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:49:02.231Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:50:02.062Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:50:02.094Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-22T14:50:02.132Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-22_07_37_18-7377215218196749982 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: edc65f7ce06c43a194f0c712a2ea5f2d and timestamp: 1598107815.9023364:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 372
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 55s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/wk7hddcz7lcf4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #440
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/440/display/redirect?page=changes>
Changes:
[david.janicek] [BEAM-10292] LocalFileSystem.matchNewResource method throws
[Kamil Wasilewski] [BEAM-10674] Python CoGBK: Remove unnecessary step. Adjust parameters to
[Kamil Wasilewski] [BEAM-10674] Python CoGBK: Add streaming job
[Kyle Weaver] [BEAM-9118] Increase portable_runner_test timeout.
[Kyle Weaver] [BEAM-10770] Remove DataflowPortabilityApiUnsupported annotation.
[daniel.o.programmer] [BEAM-3301] Consolidating starcgen method names in graph/fn.go and
[daniel.o.programmer] [BEAM-10289] Fixing miscalculated SDF split fraction.
[david.janicek] [BEAM-10292] changes after review
[noreply] [BEAM-10701] Generate Codecov XML (#12566)
[noreply] [BEAM-9979] Fix read index being reported to be reset after any metrics
[noreply] [BEAM-9546] DataframeTransform can now consume a schema-aware
------------------------------------------
[...truncated 26.89 KB...]
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting botocore<1.18.0,>=1.17.47
Using cached botocore-1.17.47-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting attrs>=17.4.0
Using cached attrs-20.1.0-py2.py3-none-any.whl (49 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting docker
Using cached docker-4.3.1-py2.py3-none-any.whl (145 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2179154 sha256=38decee90a3737b5fd7c8ce126bf38375622c2629a7099ebab81421fcdff84ac
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, idna, urllib3, chardet, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, wcwidth, attrs, packaging, more-itertools, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, deprecation, blindspin, websocket-client, docker, wrapt, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-20.1.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.47 botocore-1.17.47 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.1 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0821125344.1598020635.025780/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0821125344.1598020635.025780/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0821125344.1598020635.025780/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0821125344.1598020635.025780/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-21T14:37:16.559357Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-21_07_37_15-11450326444839000613'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0821125344'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-21T14:37:16.559357Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-21_07_37_15-11450326444839000613]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-21_07_37_15-11450326444839000613
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-21_07_37_15-11450326444839000613?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-21_07_37_15-11450326444839000613 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:19.448Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.256Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.300Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.359Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.402Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.467Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.503Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.531Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.569Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.603Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.636Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.658Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.690Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.781Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.851Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.914Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.951Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:20.993Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.029Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.301Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.370Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.418Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.442Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.477Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.541Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:21.596Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-21_07_37_15-11450326444839000613 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:37.677Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:47.424Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:37:47.447Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:38:29.807Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:39:27.986Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:39:28.017Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:47:02.002Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:47:02.081Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:47:02.130Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:47:02.211Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:03.762Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:03.873Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:03.948Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:04.014Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:04.044Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:54.491Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:54.541Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-21T14:49:54.597Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-21_07_37_15-11450326444839000613 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 8b8ed40169604de388612ddb1545aac5 and timestamp: 1598021407.4029233:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 376
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224970058198318
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224970058079616
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 50s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/aoebgsxlrjxyu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #439
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/439/display/redirect?page=changes>
Changes:
[chamikaramj] Include coders embedded in CombinePayload when converting x-lang
[zyichi] Allow Nexmark launcher to publish human-readable events to pubsub.
[douglas.damon] Add Filter lesson to Go SDK katas
[douglas.damon] Reformat code for readability
[yoshiki.obata] [BEAM-9980] Configure Python versions for direct test suite tasks via
[irvi.fa] [BEAM-10753] Add Slack link invitation on README
[Luke Cwik] [BEAM-10756] Fix empty pull response to not ack and to not throw
[ettarapp] clarifying unclear comments
[noreply] Update README.md
[noreply] [BEAM-3301] Adding SDF Go Dataflow translation. (#12629)
[kevinsijo] [BEAM-9919] Refining xlang types and wrappers
[noreply] [BEAM-10752] Use TestPubsubSignal in PubsubToBigqueryIT (#12625)
[douglas.damon] Change license info format in task-info.yaml
[douglas.damon] Update stepik course
[Luke Cwik] [BEAM-9979] Support a reset call that happens after the bundle is done
[noreply] [BEAM-10200] Add optional experiment to enable heap dump through the …
[kevinsijo] [BEAM-9919] xlang transforms wrapped with namespaces
[kevinsijo] [BEAM-9919] cleaning namespace.go
[kevinsijo] [BEAM-9920] Enabling artifact staging for xlang transforms
[kevinsijo] documentation(beam/sdks/go): added apache license and function docstring
[noreply] [BEAM-9547] Implement dataframes top, join, merge. (#12516)
[kevinsijo] fix(beam/sdks/go): correcting changed function name
[noreply] Merge pull request #12581 from [BEAM-10378] Add Azure Blob Storage
------------------------------------------
[...truncated 26.72 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting botocore<1.18.0,>=1.17.46
Using cached botocore-1.17.46-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2174879 sha256=97a826fe137a8164f736f998d2854ed048395883171070ae54b0e868e3454aca
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, chardet, urllib3, certifi, idna, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, wcwidth, atomicwrites, packaging, attrs, more-itertools, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, deprecation, wrapt, colorama, crayons, websocket-client, docker, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.46 botocore-1.17.46 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0820125344.1597934236.608705/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0820125344.1597934236.608705/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0820125344.1597934236.608705/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0820125344.1597934236.608705/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-20T14:37:18.196273Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-20_07_37_17-17211290841274263071'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0820125344'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-20T14:37:18.196273Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-20_07_37_17-17211290841274263071]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-20_07_37_17-17211290841274263071
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-20_07_37_17-17211290841274263071?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-20_07_37_17-17211290841274263071 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:20.687Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.671Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.701Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.728Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.758Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.817Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.860Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.890Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.918Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.956Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:21.995Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.034Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.067Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.092Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.122Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.158Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.205Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.243Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.273Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.395Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.453Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.501Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.525Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.591Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.660Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:22.734Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-20_07_37_17-17211290841274263071 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:33.311Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:37:50.287Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:39:32.776Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:39:32.815Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:46:55.235Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:46:55.296Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:46:55.415Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:46:55.504Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.252Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.319Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.387Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.432Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:48:56.455Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:49:48.696Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:49:48.742Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-20T14:49:48.768Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-20_07_37_17-17211290841274263071 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 798a6aff9d484bb391028126939e25da and timestamp: 1597934999.0144124:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 366
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969971790948
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969971672816
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 42s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/inmmtd2jqtwxy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #438
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/438/display/redirect?page=changes>
Changes:
[Kyle Weaver] [BEAM-9956] Remove unbalanced code markup.
[zyichi] Bump up advanceProcessingTime duration in ParDoTest
[tobiasz.kedzierski] [BEAM-10623] Add workflow to run python tests on Linux/Windows/Mac
[tobiasz.kedzierski] [BEAM-10624] dtype explicit for the numpy arrays
[noreply] [BEAM-10715] Update Jet Runner validates runner testing (#12567)
[noreply] [BEAM-10557] Implemented SchemaIOProvider for DataStoreV1, Refactored
[Steve Niemitz] [BEAM-10523] Add support for custom DatumWriters to AvroIO.Write
[Luke Cwik] [BEAM-10040, BEAM-6804] Increase wait times to reduce flakiness.
[noreply] [BEAM-10752] add use_deprecated_read experiment for testSimpleInsert,
[ekirpichov] Add enum34 to manual_licenses
[tobiasz.kedzierski] [BEAM-BEAM-10667] Make github actions bulding wheels bucket name stored
[noreply] Merge pull request #12597: [BEAM-10685] Added integration test for
------------------------------------------
[...truncated 26.74 KB...]
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting botocore<1.18.0,>=1.17.45
Using cached botocore-1.17.45-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172826 sha256=68bd9d4a99b4983341eadc3d4077e7c78c250a380ab28047bc2abb0b1dc4352c
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, chardet, idna, urllib3, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, attrs, wcwidth, more-itertools, packaging, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, wrapt, websocket-client, docker, colorama, crayons, deprecation, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.45 botocore-1.17.45 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.27.2 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0819125407.1597847847.824123/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0819125407.1597847847.824123/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0819125407.1597847847.824123/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0819125407.1597847847.824123/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-19T14:37:29.501334Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-19_07_37_28-436199677609419665'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0819125407'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-19T14:37:29.501334Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-19_07_37_28-436199677609419665]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-19_07_37_28-436199677609419665
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-19_07_37_28-436199677609419665?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-19_07_37_28-436199677609419665 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:31.830Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.593Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.623Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.668Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.696Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.761Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.798Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.826Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.853Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.888Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.919Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.950Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:32.979Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.012Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.047Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.085Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.119Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.147Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.182Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.382Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.448Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.487Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.507Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.546Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.608Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:37:33.678Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-19_07_37_28-436199677609419665 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:38:00.262Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:38:08.473Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:39:44.033Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:39:44.074Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:47:16.440Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:47:16.573Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:47:16.612Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:47:16.685Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.234Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.297Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.380Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.434Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:49:17.461Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:50:11.129Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:50:11.160Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-19T14:50:11.188Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-19_07_37_28-436199677609419665 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 5c9423c3789a47ecab88e533d660fde7 and timestamp: 1597848620.3681748:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 373
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969885411772
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969885294068
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 3s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/ua5rjgl6p36u2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #437
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/437/display/redirect?page=changes>
Changes:
[yoshiki.obata] [BEAM-9980] add groovy functions for python versions
[yoshiki.obata] [BEAM-9980] update dataflow test-suites to switch python versions using
[heejong] [BEAM-10679] improving XLang KafkaIO streaming test
[heejong] fix formatting
[heejong] fix incorrect coder issue
[heejong] use Create
[yoshiki.obata] [BEAM-9980] version switchable dataflow tasks to be invoked
[yoshiki.obata] [BEAM-9980] :sdks:python:test-suites:dataflow included in
[tobiasz.kedzierski] Improve CI documentation
[Luke Cwik] [BEAM-10670] Make key coder deterministic by using upstream PCollection
[Udi Meiri] [BEAM-10697] Remove testPy2Cython from precommit
[noreply] [BEAM-9891] Generate query execution summary table after finishing jobs
[noreply] [BEAM-9919] Added an External Transform API to Go SDK (#12445)
------------------------------------------
[...truncated 24.38 KB...]
Collecting mock<3.0.0,>=1.0.1
Using cached mock-2.0.0-py2.py3-none-any.whl (56 kB)
Collecting numpy<2,>=1.14.3
Using cached numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)
Collecting pymongo<4.0.0,>=3.8.0
Using cached pymongo-3.11.0-cp37-cp37m-manylinux2014_x86_64.whl (511 kB)
Processing /home/jenkins/.cache/pip/wheels/86/73/7a/3b3f76a2142176605ff38fbca574327962c71e25a43197a4c1/oauth2client-3.0.0-py3-none-any.whl
Requirement already satisfied: protobuf<4,>=3.12.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from apache-beam==2.25.0.dev0) (3.13.0)
Collecting pydot<2,>=1.2.0
Using cached pydot-1.4.1-py2.py3-none-any.whl (19 kB)
Collecting python-dateutil<3,>=2.8.0
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz>=2018.3
Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting requests<3.0.0,>=2.24.0
Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting typing-extensions<3.8.0,>=3.7.0
Using cached typing_extensions-3.7.4.2-py3-none-any.whl (22 kB)
Processing /home/jenkins/.cache/pip/wheels/bc/49/5f/fdb5b9d85055c478213e0158ac122b596816149a02d82e0ab1/avro_python3-1.9.2.1-py3-none-any.whl
Collecting pyarrow<0.18.0,>=0.15.1
Using cached pyarrow-0.17.1-cp37-cp37m-manylinux2014_x86_64.whl (63.8 MB)
Collecting boto3>=1.9
Using cached boto3-1.14.44-py2.py3-none-any.whl (129 kB)
Collecting cachetools<4,>=3.1.0
Using cached cachetools-3.1.1-py2.py3-none-any.whl (11 kB)
Processing /home/jenkins/.cache/pip/wheels/19/b5/2f/1cc3cf2b31e7a9cd1508731212526d9550271274d351c96f16/google_apitools-0.5.31-py3-none-any.whl
Collecting google-auth<2,>=1.18.0
Using cached google_auth-1.20.1-py2.py3-none-any.whl (91 kB)
Collecting google-cloud-datastore<2,>=1.7.1
Using cached google_cloud_datastore-1.15.0-py2.py3-none-any.whl (132 kB)
Collecting google-cloud-pubsub<2,>=0.39.0
Using cached google_cloud_pubsub-1.7.0-py2.py3-none-any.whl (144 kB)
Collecting google-cloud-bigquery<2,>=1.6.0
Using cached google_cloud_bigquery-1.26.1-py2.py3-none-any.whl (170 kB)
Collecting google-cloud-core<2,>=0.28.1
Using cached google_cloud_core-1.4.1-py2.py3-none-any.whl (26 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.19-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting botocore<1.18.0,>=1.17.44
Using cached botocore-1.17.44-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172471 sha256=aa380437a69d7ed97add617c86e0a607bb166abd7088ed7d58a902181d7c3b36
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, urllib3, chardet, certifi, idna, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, attrs, more-itertools, packaging, wcwidth, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, colorama, crayons, websocket-client, docker, blindspin, wrapt, deprecation, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.44 botocore-1.17.44 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.19 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0818141653.1597761439.456857/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0818141653.1597761439.456857/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0818141653.1597761439.456857/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0818141653.1597761439.456857/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--iterations=4', '--fanout=1']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--iterations=4', '--fanout=1']
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 150, in run
self.result = self.pipeline.run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/test_pipeline.py",> line 112, in run
False if self.not_use_test_runner_api else test_runner_api))
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/pipeline.py",> line 521, in run
allow_proto_holders=True).run(False)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/pipeline.py",> line 534, in run
return self.runner.run_pipeline(self, self._options)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 592, in run_pipeline
self.dataflow_client.create_job(self.job), self)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/utils/retry.py",> line 236, in wrapper
return fun(*args, **kwargs)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",> line 681, in create_job
return self.submit_job_description(job)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/utils/retry.py",> line 236, in wrapper
return fun(*args, **kwargs)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",> line 748, in submit_job_description
response = self._client.projects_locations_jobs.Create(request)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",> line 667, in Create
config, request, global_params=global_params)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages/apitools/base/py/base_api.py",> line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages/apitools/base/py/base_api.py",> line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages/apitools/base/py/base_api.py",> line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpConflictError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Tue, 18 Aug 2020 14:37:20 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '409', 'content-length': '334', '-content-encoding': 'gzip'}>, content <{
"error": {
"code": 409,
"message": "(569405d57578a32d): The workflow could not be created. Causes: (74dcf4736e5d95f0): There is already an active job named load-tests-python-dataflow-batch-gbk-6-0818141653. If you want to submit a second job, try again by setting a different name.",
"status": "ALREADY_EXISTS"
}
}
>
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1m 3s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/g6wqoojyi5q6q
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #436
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/436/display/redirect?page=changes>
Changes:
[katarzyna.kucharczyk] [BEAM-10672] Added streaming option to combine test
[katarzyna.kucharczyk] [BEAM-10672] Added experimental dataflow param
[katarzyna.kucharczyk] [BEAM-10672] Spottless Apply
[katarzyna.kucharczyk] [BEAM-10672] Updated streaming commands
[katarzyna.kucharczyk] [BEAM-10672] Fixes after review in combine python load test
[katarzyna.kucharczyk] [BEAM-10672] Changed jobType variable into mode to declare if it is
------------------------------------------
[...truncated 26.88 KB...]
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting botocore<1.18.0,>=1.17.43
Using cached botocore-1.17.43-py2.py3-none-any.whl (6.5 MB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172406 sha256=7f4bbd3f156da8ad4398f14f2560722eca1378e5bbb72a634fce431879f96465
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, certifi, urllib3, chardet, idna, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, rsa, pyasn1-modules, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, docutils, jmespath, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, atomicwrites, packaging, wcwidth, attrs, more-itertools, pytest, pytest-forked, apipkg, execnet, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, blindspin, deprecation, websocket-client, docker, wrapt, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.43 botocore-1.17.43 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0817125556.1597675048.243259/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0817125556.1597675048.243259/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0817125556.1597675048.243259/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0817125556.1597675048.243259/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-17T14:37:29.888404Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-17_07_37_28-2818844421715832941'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0817125556'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-17T14:37:29.888404Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-17_07_37_28-2818844421715832941]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-17_07_37_28-2818844421715832941
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-17_07_37_28-2818844421715832941?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-17_07_37_28-2818844421715832941 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:32.655Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.520Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.557Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.594Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.623Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.714Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.758Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.794Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.817Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.851Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.884Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.910Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.944Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:33.979Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.029Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.069Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.103Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.134Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.172Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.301Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.367Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.416Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.451Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.495Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.556Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:37:34.625Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-17_07_37_28-2818844421715832941 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:38:01.777Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 4 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:38:01.813Z: JOB_MESSAGE_DETAILED: Resized **** pool to 4, though goal was 5. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:38:04.107Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:38:07.184Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:39:39.058Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:39:39.085Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:47:15.734Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:47:15.783Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:47:15.820Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:47:15.897Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.605Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.688Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.754Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.802Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:49:10.830Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:50:03.569Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:50:03.603Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-17T14:50:03.637Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-17_07_37_28-2818844421715832941 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 1284ec679e944bcab07b3610e7c79db7 and timestamp: 1597675815.6649678:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 366
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969712605214
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969712493324
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 53s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/qjspdkfrricgc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #435
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/435/display/redirect?page=changes>
Changes:
[Boyuan Zhang] Remove redundant setMaxNumRecords and consumerFactoryFn.
[Luke Cwik] [BEAM-10670] Make Read execute as a splittable DoFn by default for the
[noreply] Merge pull request #12575: [BEAM-10707] Adding Python docs precommit as
------------------------------------------
[...truncated 26.65 KB...]
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.4.0-py2.py3-none-any.whl (265 kB)
Collecting google-cloud-spanner<2,>=1.13.0
Using cached google_cloud_spanner-1.17.1-py2.py3-none-any.whl (250 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting botocore<1.18.0,>=1.17.43
Using cached botocore-1.17.43-py2.py3-none-any.whl (6.5 MB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.2-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172406 sha256=0057d3a57236d954c1f6a388cc2f7fb9944cd9f2a1a2abe403d030017467df00
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Successfully built apache-beam
Installing collected packages: crcmod, dill, pytz, fastavro, docopt, urllib3, idna, chardet, certifi, requests, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, more-itertools, packaging, attrs, atomicwrites, wcwidth, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, blindspin, colorama, crayons, websocket-client, docker, deprecation, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.43 botocore-1.17.43 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0816125613.1597588647.925574/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0816125613.1597588647.925574/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0816125613.1597588647.925574/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0816125613.1597588647.925574/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-16T14:37:29.630461Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-16_07_37_28-6918673436260540560'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0816125613'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-16T14:37:29.630461Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-16_07_37_28-6918673436260540560]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-16_07_37_28-6918673436260540560
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-16_07_37_28-6918673436260540560?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-16_07_37_28-6918673436260540560 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:32.161Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:32.934Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:32.968Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.003Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.036Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.094Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.125Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.157Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.189Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.212Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.248Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.279Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.305Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.325Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.347Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.382Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.416Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.450Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.475Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.633Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.692Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.737Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.771Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.812Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.860Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:33.916Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-16_07_37_28-6918673436260540560 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:37:58.866Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:38:04.664Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:39:41.884Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:39:41.915Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:47:04.103Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:47:04.185Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:47:04.268Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:47:04.321Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.567Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.628Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.705Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.770Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:49:02.802Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:50:04.731Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:50:04.767Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-16T14:50:04.795Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-16_07_37_28-6918673436260540560 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 20aa44a18907477294df4d938d1d0745 and timestamp: 1597589415.4301322:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 366
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 55s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/x2krlh44thuxw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins:
beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch #434
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/434/display/redirect?page=changes>
Changes:
[Luke Cwik] [BEAM-10688] Euphoria assumes that all type descriptors are resolvable
[Luke Cwik] [BEAM-10670] Use fraction of remainder if consumed fraction is unknown
[Luke Cwik] [BEAM-10670] Improve splitting logic to prefer splits upto the the
[Luke Cwik] [BEAM-10670] Fix passing forward the self-checkpoint from the
[Boyuan Zhang] Enable dataflow streaming engine when running runner_v2 and streaming.
[Boyuan Zhang] Fix formatter.
[Boyuan Zhang] Use unbounded wrapper for Kafka Read.
[je.ik] [BEAM-10691] Use FlinkStateInternals#addWatermarkHoldUsage for timer
[noreply] [BEAM-9615] Map user types to Schema reps. (#12554)
[noreply] fixed a typo in S3TestUtils (#12582)
[neville.lyh] [BEAM-10612] Add flink 1.11 runner
[Boyuan Zhang] Scale progress with respect to windows observation.
[noreply] [BEAM-9547] Implement some methods for deferred Series. (#12534)
[Robert Burke] Fix broken build.
[Luke Cwik] [BEAM-8025] Update tests to use TemporaryFolder instead of rolling their
------------------------------------------
[...truncated 26.93 KB...]
Collecting google-cloud-dlp<2,>=0.12.0
Using cached google_cloud_dlp-1.0.0-py2.py3-none-any.whl (169 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.15.0-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.0-py2.py3-none-any.whl (435 kB)
Collecting freezegun>=0.3.12
Using cached freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
Collecting nose>=1.3.7
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Processing /home/jenkins/.cache/pip/wheels/2e/29/a9/431158315f33abeaad2905460f5ffb18fc30f7ed7c66c47dee/nose_xunitmp-0.4.1-py3-none-any.whl
Collecting parameterized<0.8.0,>=0.7.1
Using cached parameterized-0.7.4-py2.py3-none-any.whl (25 kB)
Collecting pyhamcrest!=1.10.0,<2.0.0,>=1.9
Using cached PyHamcrest-1.10.1-py3-none-any.whl (48 kB)
Processing /home/jenkins/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653/PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting requests_mock<2.0,>=1.7
Using cached requests_mock-1.8.0-py2.py3-none-any.whl (23 kB)
Collecting tenacity<6.0,>=5.0.2
Using cached tenacity-5.1.5-py2.py3-none-any.whl (34 kB)
Collecting pytest<5.0,>=4.4.0
Using cached pytest-4.6.11-py2.py3-none-any.whl (231 kB)
Collecting pytest-xdist<2,>=1.29.0
Using cached pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)
Collecting pytest-timeout<2,>=1.3.3
Using cached pytest_timeout-1.4.2-py2.py3-none-any.whl (10 kB)
Collecting pandas<1,>=0.25.2
Using cached pandas-0.25.3-cp37-cp37m-manylinux1_x86_64.whl (10.4 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.3.18-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.8.5-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Processing /home/jenkins/.cache/pip/wheels/22/11/6e/126cbd2f544f9966ee22e074e5a3b7ab9b22bb2d4a6217e599/testcontainers-3.0.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from grpcio<2,>=1.29.0->apache-beam==2.25.0.dev0) (1.15.0)
Processing /home/jenkins/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Collecting pbr>=0.11
Using cached pbr-5.4.5-py2.py3-none-any.whl (110 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules>=0.0.5
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa>=3.1.4
Using cached rsa-4.6-py3-none-any.whl (47 kB)
Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from protobuf<4,>=3.12.2->apache-beam==2.25.0.dev0) (49.6.0)
Collecting pyparsing>=2.1.4
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting botocore<1.18.0,>=1.17.43
Using cached botocore-1.17.43-py2.py3-none-any.whl (6.5 MB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
Using cached s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.14.0
Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
Processing /home/jenkins/.cache/pip/wheels/b9/ee/67/2e444183030cb8d31ce8b34cee34a7afdbd3ba5959ea846380/grpc_google_iam_v1-0.12.3-py3-none-any.whl
Collecting google-resumable-media<2.0dev,>=0.5.0
Using cached google_resumable_media-0.7.1-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8" in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.7.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0; python_version > "2.7"
Using cached more_itertools-8.4.0-py3-none-any.whl (43 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (1.9.0)
Collecting attrs>=17.4.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: pluggy<1.0,>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (0.13.1)
Collecting packaging
Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting execnet>=1.1
Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.3.0-py2.py3-none-any.whl (4.7 kB)
Processing /home/jenkins/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6/wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl
Collecting docker
Using cached docker-4.3.0-py2.py3-none-any.whl (145 kB)
Processing /home/jenkins/.cache/pip/wheels/4e/29/41/2a34b96f3d4bf9ab8ccaac5d86db4334fa17d79e7027a5b86f/blindspin-2.0.1-py3-none-any.whl
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting crayons
Using cached crayons-0.3.1-py2.py3-none-any.whl (4.6 kB)
Collecting docutils<0.16,>=0.10
Using cached docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting monotonic>=0.1
Using cached monotonic-1.5-py2.py3-none-any.whl (5.3 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Collecting google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"
Using cached google_crc32c-0.1.0-cp37-cp37m-manylinux2010_x86_64.whl (39 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12; python_version < "3.8"->pytest<5.0,>=4.4.0->apache-beam==2.25.0.dev0) (3.1.0)
Collecting apipkg>=1.4
Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting colorama
Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting cffi>=1.0.0
Using cached cffi-1.14.1-cp37-cp37m-manylinux1_x86_64.whl (401 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: apache-beam, boto3
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.25.0.dev0-py3-none-any.whl size=2172406 sha256=7714d31a5149788b6cea204fd12ba1f12209a8714bf9fe57db5b610bcdb6d9fd
Stored in directory: /home/jenkins/.cache/pip/wheels/7b/59/6b/8a29b310e3f8232e5e9f8ba3c9c5d92a4bf2dc03051999ab40
Building wheel for boto3 (setup.py): started
Building wheel for boto3 (setup.py): finished with status 'done'
Created wheel for boto3: filename=boto3-1.14.43-py2.py3-none-any.whl size=127852 sha256=683cbfa20b1dce577c795ed52d32340a84770e610811ea746df0bad2db15da10
Stored in directory: /home/jenkins/.cache/pip/wheels/2a/2a/a0/48b976bfdd536c39baf38c9f11206ccaad257284eb7cd4aeb2
Successfully built apache-beam boto3
Installing collected packages: crcmod, dill, pytz, fastavro, certifi, urllib3, chardet, idna, requests, docopt, hdfs, httplib2, pbr, mock, numpy, pymongo, pyasn1, pyasn1-modules, rsa, oauth2client, pyparsing, pydot, python-dateutil, typing-extensions, avro-python3, pyarrow, jmespath, docutils, botocore, s3transfer, boto3, cachetools, monotonic, fasteners, google-apitools, google-auth, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-datastore, grpc-google-iam-v1, google-cloud-pubsub, pycparser, cffi, google-crc32c, google-resumable-media, google-cloud-bigquery, google-cloud-bigtable, google-cloud-spanner, grpcio-gcp, google-cloud-dlp, google-cloud-language, google-cloud-videointelligence, google-cloud-vision, freezegun, nose, nose-xunitmp, parameterized, pyhamcrest, pyyaml, requests-mock, tenacity, wcwidth, atomicwrites, more-itertools, attrs, packaging, pytest, apipkg, execnet, pytest-forked, pytest-xdist, pytest-timeout, pandas, sqlalchemy, psycopg2-binary, wrapt, websocket-client, docker, blindspin, deprecation, colorama, crayons, testcontainers, apache-beam
Successfully installed apache-beam-2.25.0.dev0 apipkg-1.5 atomicwrites-1.4.0 attrs-19.3.0 avro-python3-1.9.2.1 blindspin-2.0.1 boto3-1.14.43 botocore-1.17.43 cachetools-3.1.1 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 colorama-0.4.3 crayons-0.3.1 crcmod-1.7 deprecation-2.1.0 dill-0.3.1.1 docker-4.3.0 docopt-0.6.2 docutils-0.15.2 execnet-1.7.1 fastavro-0.23.6 fasteners-0.15 freezegun-0.3.15 google-api-core-1.22.1 google-apitools-0.5.31 google-auth-1.20.1 google-cloud-bigquery-1.26.1 google-cloud-bigtable-1.4.0 google-cloud-core-1.4.1 google-cloud-datastore-1.15.0 google-cloud-dlp-1.0.0 google-cloud-language-1.3.0 google-cloud-pubsub-1.7.0 google-cloud-spanner-1.17.1 google-cloud-videointelligence-1.15.0 google-cloud-vision-1.0.0 google-crc32c-0.1.0 google-resumable-media-0.7.1 googleapis-common-protos-1.52.0 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 hdfs-2.5.8 httplib2-0.17.4 idna-2.10 jmespath-0.10.0 mock-2.0.0 monotonic-1.5 more-itertools-8.4.0 nose-1.3.7 nose-xunitmp-0.4.1 numpy-1.19.1 oauth2client-3.0.0 packaging-20.4 pandas-0.25.3 parameterized-0.7.4 pbr-5.4.5 psycopg2-binary-2.8.5 pyarrow-0.17.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydot-1.4.1 pyhamcrest-1.10.1 pymongo-3.11.0 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.3.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.24.0 requests-mock-1.8.0 rsa-4.6 s3transfer-0.3.3 sqlalchemy-1.3.18 tenacity-5.1.5 testcontainers-3.0.3 typing-extensions-3.7.4.2 urllib3-1.25.10 wcwidth-0.2.5 websocket-client-0.57.0 wrapt-1.12.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0815125556.1597502249.024011/pipeline.pb...
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0815125556.1597502249.024011/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0815125556.1597502249.024011/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-6-0815125556.1597502249.024011/dataflow_python_sdk.tar in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--fanout=1', '--iterations=4']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
createTime: '2020-08-15T14:37:30.711981Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-15_07_37_29-7011471637501784593'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-gbk-6-0815125556'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-15T14:37:30.711981Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2020-08-15_07_37_29-7011471637501784593]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2020-08-15_07_37_29-7011471637501784593
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-15_07_37_29-7011471637501784593?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-15_07_37_29-7011471637501784593 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:33.261Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:33.975Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.012Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step GroupByKey 0: GroupByKey not followed by a combiner.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.054Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.095Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.168Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.213Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.252Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: Start into Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.288Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Measure time: Start
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.321Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Reify into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.355Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/Write into GroupByKey 0/Reify
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.392Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupByKey 0/GroupByWindow into GroupByKey 0/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.427Z: JOB_MESSAGE_DETAILED: Fusing consumer Ungroup 0 into GroupByKey 0/GroupByWindow
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.464Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure latency into Ungroup 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.500Z: JOB_MESSAGE_DETAILED: Fusing consumer Measure time: End 0 into Measure latency
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.539Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.572Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.620Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.656Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.907Z: JOB_MESSAGE_DEBUG: Executing wait step start13
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:34.979Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.024Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.059Z: JOB_MESSAGE_BASIC: Starting 5 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.104Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Create
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.175Z: JOB_MESSAGE_DEBUG: Value "GroupByKey 0/Session" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:35.245Z: JOB_MESSAGE_BASIC: Executing operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-15_07_37_29-7011471637501784593 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:37:59.720Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:38:05.392Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:39:38.341Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:39:38.383Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:47:02.391Z: JOB_MESSAGE_BASIC: Finished operation Read+Measure time: Start+Assign timestamps+GroupByKey 0/Reify+GroupByKey 0/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:47:02.481Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:47:02.532Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:47:02.595Z: JOB_MESSAGE_BASIC: Executing operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:02.932Z: JOB_MESSAGE_BASIC: Finished operation GroupByKey 0/Read+GroupByKey 0/GroupByWindow+Ungroup 0+Measure latency+Measure time: End 0
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:03.010Z: JOB_MESSAGE_DEBUG: Executing success step success11
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:03.087Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:03.159Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:03.193Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:57.397Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:57.441Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-08-15T14:49:57.472Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-08-15_07_37_29-7011471637501784593 is in state JOB_STATE_DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 71154216039e4a1d8ccda4f841759318 and timestamp: 1597503011.5470157:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 367
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_count_latency Value: 400
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_max_latency Value: 9224969539797436
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_min_latency Value: 9224969539679924
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py",> line 120, in <module>
GroupByKeyTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 153, in run
self._metrics_monitor.publish_metrics(self.result)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 227, in publish_metrics
publisher.publish(insert_dicts)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py",> line 375, in publish
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
TypeError: %d format: a number is required, not NoneType
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 13m 52s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/qoktkzn32do5a
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org