You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2022/05/02 20:01:14 UTC
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #571
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/571/display/redirect?page=changes>
Changes:
[noreply] [BEAM-11105] Add manual watermark estimation (#17475)
------------------------------------------
[...truncated 328.48 KB...]
Using cached testcontainers-3.5.3-py2.py3-none-any.whl (35 kB)
Collecting cryptography>=36.0.0
Using cached cryptography-37.0.1-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.39.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting botocore<1.26.0,>=1.25.4
Using cached botocore-1.25.4-py3-none-any.whl (8.7 MB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.39.0.dev0) (21.3)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.39.0.dev0) (62.1.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.44.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.8.1-py2.py3-none-any.whl (113 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.12.0-py3-none-any.whl (54 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.39.0.dev0) (1.11.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.39.0.dev0) (2.1.3)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.39.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.39.0.dev0-py3-none-any.whl size=2737178 sha256=0e968f15104525e987427d93dd9e53e0b5cc806372404ee426b4301448e2bf0f
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.8
Uninstalling pyparsing-3.0.8:
Successfully uninstalled pyparsing-3.0.8
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.39.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.23.1 azure-storage-blob-12.11.0 boto3-1.22.4 botocore-1.25.4 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.1 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.44.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.12.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.8.1 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.39.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fc3c07abcb0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fc3c07ac4d0> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651506577.749512/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651506577.749512/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651506577.749512/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651506577.749512/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220502154937750454-6199'
createTime: '2022-05-02T15:49:38.841702Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-02_08_49_38-6566892061520285346'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0428111049'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-02T15:49:38.841702Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-02_08_49_38-6566892061520285346]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-02_08_49_38-6566892061520285346
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-02_08_49_38-6566892061520285346?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-02_08_49_38-6566892061520285346 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:45.469Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.481Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.543Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.576Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.603Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.639Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.668Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.689Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.711Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.734Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.757Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.787Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.811Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.844Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.879Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:46.903Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:47.043Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:47.103Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:47.204Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:49:47.262Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-02_08_49_38-6566892061520285346 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:50:09.333Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:50:18.976Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:50:42.190Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:56:55.971Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:56:56.046Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:56:56.070Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:56:56.142Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:56:56.187Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:56:56.245Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T15:56:56.315Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T20:00:30.268Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-02_08_49_38-6566892061520285346.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T20:00:30.296Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T20:00:30.452Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T20:00:30.549Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T20:00:30.583Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-02_08_49_38-6566892061520285346 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T20:01:02.662Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T20:01:02.702Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-02T20:01:02.728Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-02_08_49_38-6566892061520285346 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1661, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 12m 36s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/uwckbjskahiss
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #586
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/586/display/redirect?page=changes>
Changes:
[noreply] [BEAM-13015] Update the SDK harness grouping table to be memory bounded
[noreply] [BEAM-13982] Added output of logging for python E2E pytests (#17637)
[noreply] [BEAM-14473] Throw error if using globally windowed, unbounded side
[noreply] [BEAM-14440] Add basic fuzz tests to the coders package (#17587)
[noreply] [BEAM-14035 ] Implement BigQuerySchema Read/Write TransformProvider
[noreply] Add Akvelon to case-studies (#17611)
[noreply] Merge pull request #17520 from BEAM-12356 Close DatasetService leaked
[noreply] Adding eslint and lint configuration to TypeScript SDK (#17676)
[noreply] Update release-guide.md
[noreply] Update release-guide.md
[noreply] [BEAM-14411] Re-enable TypecodersTest, fix most issues (#17547)
[noreply] Merge pull request #17678 from [BEAM-14460] [Playground] WIP. Fix error
------------------------------------------
[...truncated 329.35 KB...]
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.3.1)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.1-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.21.0,>=0.8->apache-beam==2.40.0.dev0) (3.0.9)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker>=4.0.0
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.1-py2.py3-none-any.whl (211 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2776974 sha256=82df37f504bf12ee0d94d26c7677e98177e8dfb8f52f7ef78f8388b17215f237
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pymysql, pymongo, pyhamcrest, pydot, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, httplib2, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pyarrow, pluggy, pandas, overrides, oauth2client, mock, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, deprecation, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-apitools, google-api-core, docker, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.12.0 boto3-1.23.1 botocore-1.26.1 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.1 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.1 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.1 hdfs-2.7.0 httplib2-0.20.4 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.4 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7ffa90107e60> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7ffa90106680> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0517100510.1652802904.470451/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0517100510.1652802904.470451/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0517100510.1652802904.470451/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0517100510.1652802904.470451/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220517155504471861-6359'
createTime: '2022-05-17T15:55:05.708475Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-17_08_55_05-5019473855042774752'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0517100510'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-17T15:55:05.708475Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-17_08_55_05-5019473855042774752]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-17_08_55_05-5019473855042774752
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-17_08_55_05-5019473855042774752?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-17_08_55_05-5019473855042774752 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:09.189Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:10.641Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:10.846Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.021Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.050Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.104Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.138Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.185Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.238Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.267Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.305Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.329Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.376Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.413Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.446Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.482Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.673Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.762Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.823Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:11.862Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-17_08_55_05-5019473855042774752 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:19.435Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:55:35.214Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T15:56:10.291Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T16:03:04.621Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T16:03:04.733Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T16:03:04.775Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T16:03:04.849Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T16:03:04.910Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T16:03:05.001Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T16:03:05.066Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T20:00:52.635Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-17_08_55_05-5019473855042774752.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T20:00:52.688Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-17_08_55_05-5019473855042774752 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T20:00:52.804Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T20:00:53.230Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T20:00:53.254Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T20:01:35.180Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T20:01:35.222Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-17T20:01:35.248Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-17_08_55_05-5019473855042774752 is in state JOB_STATE_CANCELLED
ERROR:apache_beam.runners.dataflow.dataflow_runner:Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-05-17_08_55_05-5019473855042774752?project=<ProjectId>
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1675, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 7m 57s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
Publishing failed.
The build scan server appears to be unavailable.
Please check https://status.gradle.com for the latest service status.
If the service is reported as available, please report this problem via https://gradle.com/help/plugin and include the following via copy/paste:
----------
Gradle version: 7.4
Plugin version: 3.4.1
Request URL: https://status.gradle.com
Request ID: 98ea2449-f176-4e6e-877c-de4227dfd3fb
Response status code: 405
Response server type: Varnish
----------
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #585
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/585/display/redirect>
Changes:
------------------------------------------
[...truncated 328.64 KB...]
Using cached google_cloud_language-1.3.1-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.16.2-py2.py3-none-any.whl (183 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.1-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-recommendations-ai<=0.2.0,>=0.1.0
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting botocore<1.27.0,>=1.26.0
Using cached botocore-1.26.0-py3-none-any.whl (8.7 MB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.2.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.1-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.21.0,>=0.8->apache-beam==2.40.0.dev0) (3.0.9)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting docker>=4.0.0
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.1-py2.py3-none-any.whl (211 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2777028 sha256=2a147f8ee8d5600482d312734293b0df21ebd53a41d41aaa6b8641615c1a9f34
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pymysql, pymongo, pyhamcrest, pydot, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, httplib2, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pyarrow, pluggy, pandas, overrides, oauth2client, mock, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, deprecation, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-apitools, google-api-core, docker, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.12.0 boto3-1.23.0 botocore-1.26.0 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.1 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.1 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.1 hdfs-2.7.0 httplib2-0.20.4 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.4 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f21a6d229e0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f21a6d24200> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0516100416.1652714625.844294/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0516100416.1652714625.844294/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0516100416.1652714625.844294/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0516100416.1652714625.844294/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220516152345845272-4873'
createTime: '2022-05-16T15:23:46.839913Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-16_08_23_46-3157782307257617332'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0516100416'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-16T15:23:46.839913Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-16_08_23_46-3157782307257617332]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-16_08_23_46-3157782307257617332
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-16_08_23_46-3157782307257617332?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-16_08_23_46-3157782307257617332 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:48.730Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-c.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.639Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.669Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.722Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.750Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.787Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.813Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.874Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.907Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.940Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:49.996Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:50.027Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:50.064Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:50.095Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:50.129Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:50.165Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:50.291Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:50.422Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:50.457Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:23:50.486Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-c...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-16_08_23_46-3157782307257617332 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:24:13.067Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:24:24.644Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:24:46.370Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:30:25.154Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:30:25.213Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:30:25.245Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:30:25.345Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:30:25.395Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:30:25.461Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T15:30:25.515Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T20:00:45.296Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-16_08_23_46-3157782307257617332.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T20:00:45.318Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T20:00:45.491Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T20:00:45.524Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T20:00:45.540Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-16_08_23_46-3157782307257617332 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T20:01:19.927Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T20:01:19.961Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-16T20:01:19.993Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-16_08_23_46-3157782307257617332 is in state JOB_STATE_CANCELLED
ERROR:apache_beam.runners.dataflow.dataflow_runner:Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-05-16_08_23_46-3157782307257617332?project=<ProjectId>
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1675, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 38m 53s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/3zddelj3nsvxe
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #584
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/584/display/redirect?page=changes>
Changes:
[noreply] [BEAM-14470] Use Generic Registrations in loadtests. (#17673)
------------------------------------------
[...truncated 327.94 KB...]
Using cached google_cloud_videointelligence-1.16.2-py2.py3-none-any.whl (183 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.1-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-recommendations-ai<=0.2.0,>=0.1.0
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Collecting boto3>=1.9
Using cached boto3-1.23.0-py3-none-any.whl (132 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting botocore<1.27.0,>=1.26.0
Using cached botocore-1.26.0-py3-none-any.whl (8.7 MB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.2.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.1-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.21.0,>=0.8->apache-beam==2.40.0.dev0) (3.0.9)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting docker>=4.0.0
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.1-py2.py3-none-any.whl (211 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2777028 sha256=857d0b403fba2be07794f04ab5e5818526a3979cad5b57383f1d6298427a2a4d
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pymysql, pymongo, pyhamcrest, pydot, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, httplib2, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pyarrow, pluggy, pandas, overrides, oauth2client, mock, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, deprecation, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-apitools, google-api-core, docker, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.12.0 boto3-1.23.0 botocore-1.26.0 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.1 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.1 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.1 hdfs-2.7.0 httplib2-0.20.4 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.4 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f38538299e0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f385382a200> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0515100431.1652630227.134334/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0515100431.1652630227.134334/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0515100431.1652630227.134334/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0515100431.1652630227.134334/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220515155707135305-6643'
createTime: '2022-05-15T15:57:08.049483Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-15_08_57_07-752468079755972449'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0515100431'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-15T15:57:08.049483Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-15_08_57_07-752468079755972449]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-15_08_57_07-752468079755972449
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-15_08_57_07-752468079755972449?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-15_08_57_07-752468079755972449 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:10.267Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:11.650Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:11.786Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:11.831Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:11.867Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.016Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.072Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.134Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.192Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.235Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.284Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.333Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.425Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.460Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.507Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.532Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.646Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.733Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.815Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:12.849Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-15_08_57_07-752468079755972449 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:35.145Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:57:42.038Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T15:58:08.550Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T16:04:52.380Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T16:04:52.483Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T16:04:52.505Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T16:04:52.575Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T16:04:52.648Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T16:04:52.717Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T16:04:52.774Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T20:00:53.924Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-15_08_57_07-752468079755972449.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T20:00:53.948Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T20:00:54.022Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T20:00:54.198Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T20:00:54.218Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-15_08_57_07-752468079755972449 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T20:01:25.693Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T20:01:25.730Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-15T20:01:25.754Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-15_08_57_07-752468079755972449 is in state JOB_STATE_CANCELLED
ERROR:apache_beam.runners.dataflow.dataflow_runner:Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-05-15_08_57_07-752468079755972449?project=<ProjectId>
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1675, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 5m 32s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/ts23nhdng7ayc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #583
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/583/display/redirect?page=changes>
Changes:
[Heejong Lee] [BEAM-14455] Add UUID to sub-schemas for PythonExternalTransform
[Heejong Lee] [BEAM-14430] Adding a logical type support for Python callables to Row
[Heejong Lee] add urn, type inference for PythonCallableSource
[Heejong Lee] fix lint errors
[Heejong Lee] move logical types def
[Heejong Lee] add micros_instant urn
[Heejong Lee] put a default type hint for PythonCallableSource
[Heejong Lee] add comment
[noreply] Revert "Better test assertion. (#17551)"
[noreply] Bump github.com/spf13/cobra from 1.3.0 to 1.4.0 in /sdks (#17647)
[noreply] [BEAM-14465] Reduce DefaultS3ClientBuilderFactory logging to debug level
[noreply] Merge pull request #17365 from [BEAM-12482] Update Schema Destination
[noreply] [BEAM-14014] Support impersonation credentials in dataflow runner
[noreply] [BEAM-14469] Allow nil primary returns from TrySplit in a single-window
[noreply] Add some auto-starting runners to the typescript SDK. (#17580)
[noreply] [BEAM-14371] (and BEAM-14372) - enable a couple staticchecks (#17670)
------------------------------------------
[...truncated 328.21 KB...]
Using cached SQLAlchemy-1.4.36-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.9.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
Collecting testcontainers[mysql]<4.0.0,>=3.0.3
Using cached testcontainers-3.5.4-py2.py3-none-any.whl (39 kB)
Collecting cryptography>=36.0.0
Using cached cryptography-37.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting botocore<1.27.0,>=1.26.0
Using cached botocore-1.26.0-py3-none-any.whl (8.7 MB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.2.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.1-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.21.0,>=0.8->apache-beam==2.40.0.dev0) (3.0.9)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting docker>=4.0.0
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.1-py2.py3-none-any.whl (211 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2777028 sha256=6dac0c1fc40532e09ca0586997dfae07747b6e1a55e9b788ca02c43867399b40
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pymysql, pymongo, pyhamcrest, pydot, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, httplib2, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pyarrow, pluggy, pandas, overrides, oauth2client, mock, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, deprecation, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-apitools, google-api-core, docker, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.12.0 boto3-1.23.0 botocore-1.26.0 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.1 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.1 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.1 hdfs-2.7.0 httplib2-0.20.4 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.4 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f599ed319e0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f599ed32200> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0514100413.1652543730.848988/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0514100413.1652543730.848988/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0514100413.1652543730.848988/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0514100413.1652543730.848988/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220514155530849937-9999'
createTime: '2022-05-14T15:55:31.712259Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-14_08_55_31-1481361689829031140'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0514100413'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-14T15:55:31.712259Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-14_08_55_31-1481361689829031140]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-14_08_55_31-1481361689829031140
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-14_08_55_31-1481361689829031140?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-14_08_55_31-1481361689829031140 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:33.721Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.390Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.432Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.463Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.514Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.546Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.576Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.605Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.637Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.697Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.737Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.763Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.859Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.919Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:34.973Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:35.044Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:35.206Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:35.286Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:35.352Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:35.394Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-14_08_55_31-1481361689829031140 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:55.733Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:55:57.959Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T15:56:30.044Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T16:03:14.581Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T16:03:14.677Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T16:03:14.708Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T16:03:14.772Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T16:03:14.825Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T16:03:14.922Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T16:03:15.001Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T20:00:41.036Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-14_08_55_31-1481361689829031140.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T20:00:41.095Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T20:00:41.255Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T20:00:41.299Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T20:00:41.390Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-14_08_55_31-1481361689829031140 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T20:01:15.693Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T20:01:15.725Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-14T20:01:15.779Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-14_08_55_31-1481361689829031140 is in state JOB_STATE_CANCELLED
ERROR:apache_beam.runners.dataflow.dataflow_runner:Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-05-14_08_55_31-1481361689829031140?project=<ProjectId>
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1675, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 6m 56s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/c6domqtmavyh2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #582
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/582/display/redirect?page=changes>
Changes:
[dannymccormick] [BEAM-14441] Add GitHub issue templates
[dannymccormick] Ask for beam version + other dependencies
[dannymccormick] We don't need outage
[dannymccormick] Cut p4
[chamikaramj] Updates CHANGES.md to include some recently discovered known issues
[noreply] [BEAM-14334] Fix leakage of SparkContext in Spark runner tests to remove
[noreply] Typo & link update (#17633)
[dannymccormick] Pare down to fewer templates
[noreply] Trigger go precommits on go mod/sum changes (#17636)
[noreply] Revert "[BEAM-14429] Force java load test on dataflow runner v2
[noreply] [BEAM-14347] Add generic registration feature to CHANGES (#17643)
[noreply] Better test assertion. (#17551)
[noreply] Bump github.com/google/go-cmp from 0.5.7 to 0.5.8 in /sdks (#17628)
[noreply] Bump github.com/testcontainers/testcontainers-go in /sdks (#17627)
[noreply] Bump github.com/lib/pq from 1.10.4 to 1.10.5 in /sdks (#17626)
[noreply] Merge pull request #17584 from [BEAM-14415] Exception handling tests and
[noreply] Bump cloud.google.com/go/pubsub from 1.18.0 to 1.21.1 in /sdks (#17646)
[noreply] Merge pull request #17408 from [BEAM-14312] [Website] change section
[noreply] Bump cloud.google.com/go/bigquery from 1.28.0 to 1.32.0 in /sdks
[noreply] [BEAM-14347] Add function for simple function registration (#17650)
[noreply] Drop dataclasses requirement, we only support python 3.7+ (#17640)
------------------------------------------
[...truncated 329.46 KB...]
Using cached google_cloud_vision-1.0.1-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-recommendations-ai<=0.2.0,>=0.1.0
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Collecting azure-storage-blob>=12.3.2
Using cached azure_storage_blob-12.12.0-py3-none-any.whl (366 kB)
Collecting azure-core>=1.7.0
Using cached azure_core-1.24.0-py3-none-any.whl (178 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting botocore<1.26.0,>=1.25.13
Using cached botocore-1.25.13-py3-none-any.whl (8.7 MB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.2.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.1-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.21.0,>=0.8->apache-beam==2.40.0.dev0) (3.0.9)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.1-py2.py3-none-any.whl (211 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2773552 sha256=4ba0c7d46acdc1bc61e8188d345995a703630678da6885735951d6c6ed4db898
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pymysql, pymongo, pyhamcrest, pydot, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, httplib2, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pyarrow, pluggy, pandas, overrides, oauth2client, mock, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, deprecation, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-apitools, google-api-core, docker, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.12.0 boto3-1.22.13 botocore-1.25.13 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.1 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.1 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.1 hdfs-2.7.0 httplib2-0.20.4 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f2ee8384320> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f2ee8384b00> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0513100445.1652456647.625782/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0513100445.1652456647.625782/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0513100445.1652456647.625782/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0513100445.1652456647.625782/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220513154407627013-3232'
createTime: '2022-05-13T15:44:09.044727Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-13_08_44_08-17450588225137531890'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0513100445'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-13T15:44:09.044727Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-13_08_44_08-17450588225137531890]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-13_08_44_08-17450588225137531890
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-13_08_44_08-17450588225137531890?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-13_08_44_08-17450588225137531890 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:33.286Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:38.673Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:38.806Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:38.857Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:38.912Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.026Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.081Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.150Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.194Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.227Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.257Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.284Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.311Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.348Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.422Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:39.485Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-13_08_44_08-17450588225137531890 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:40.737Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:41.400Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:41.695Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:41.817Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:44:44.838Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:45:16.733Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:45:41.784Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:52:20.426Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:52:20.484Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:52:20.508Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:52:20.604Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:52:20.654Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:52:20.716Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T15:52:20.775Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T20:00:41.791Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-13_08_44_08-17450588225137531890.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T20:00:42.168Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T20:00:42.488Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T20:00:42.563Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T20:00:42.590Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-13_08_44_08-17450588225137531890 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T20:01:17.797Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T20:01:18.408Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-13T20:01:18.565Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-13_08_44_08-17450588225137531890 is in state JOB_STATE_CANCELLED
ERROR:apache_beam.runners.dataflow.dataflow_runner:Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-05-13_08_44_08-17450588225137531890?project=<ProjectId>
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1675, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 18m 59s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/qay27rdwredrw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #581
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/581/display/redirect?page=changes>
Changes:
[randomstep] [BEAM-14096] bump junit-quickcheck to 1.0
[noreply] [BEAM-5492] Python Dataflow integration tests should export the pipeline
[noreply] [BEAM-14396] Bump httplib2 upper bound. (#17602)
[noreply] [BEAM-11104] Add self-checkpointing to CHANGES.md (#17612)
[noreply] [BEAM-14081] [CdapIO] Add context classes for CDAP plugins (#17104)
[noreply] [BEAM-12526] Add Dependabot (#17563)
[noreply] Remove python 3.6 postcommit from mass_comment.py (#17630)
[noreply] [BEAM-14347] Add some benchmarks for generic registration (#17613)
[noreply] Correctly route go dependency changes to go label (#17632)
[noreply] [BEAM-13695] Add jamm jvm options to Java 11 (#17178)
------------------------------------------
[...truncated 329.07 KB...]
Using cached testcontainers-3.5.3-py2.py3-none-any.whl (35 kB)
Collecting cryptography>=36.0.0
Using cached cryptography-37.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
Collecting azure-storage-blob>=12.3.2
Using cached azure_storage_blob-12.12.0-py3-none-any.whl (366 kB)
Collecting azure-core>=1.7.0
Using cached azure_core-1.24.0-py3-none-any.whl (178 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting botocore<1.26.0,>=1.25.12
Using cached botocore-1.25.12-py3-none-any.whl (8.7 MB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.2.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.1-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from httplib2<0.21.0,>=0.8->apache-beam==2.40.0.dev0) (3.0.9)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2771561 sha256=fe0c31ca953802746fc670ab44974d7bff445e72bfa65f455bece43d41413fb0
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pymysql, pymongo, pyhamcrest, pydot, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, httplib2, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pyarrow, pluggy, pandas, overrides, oauth2client, mock, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, deprecation, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-apitools, google-api-core, docker, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.12.0 boto3-1.22.12 botocore-1.25.12 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.1 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.1 hdfs-2.7.0 httplib2-0.20.4 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f4f2a36d290> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f4f2a36da70> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0512100431.1652370716.096521/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0512100431.1652370716.096521/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0512100431.1652370716.096521/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0512100431.1652370716.096521/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220512155156097440-4582'
createTime: '2022-05-12T15:51:57.230040Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-12_08_51_56-17769454670639241342'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0512100431'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-12T15:51:57.230040Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-12_08_51_56-17769454670639241342]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-12_08_51_56-17769454670639241342
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-12_08_51_56-17769454670639241342?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-12_08_51_56-17769454670639241342 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:00.146Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.050Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.087Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.115Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.187Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.226Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.274Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.308Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.378Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.447Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.534Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.618Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:01.914Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:02.028Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:02.133Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:02.225Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-12_08_51_56-17769454670639241342 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:02.467Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:02.535Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:02.582Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:02.604Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:25.382Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:38.752Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:52:58.969Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:59:42.468Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:59:42.519Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:59:42.546Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:59:42.606Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:59:42.654Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:59:42.710Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T15:59:42.774Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T20:00:39.373Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-12_08_51_56-17769454670639241342.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T20:00:39.400Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T20:00:42.912Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T20:00:43.076Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T20:00:43.113Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-12_08_51_56-17769454670639241342 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T20:01:20.948Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T20:01:20.997Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-12T20:01:21.023Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-12_08_51_56-17769454670639241342 is in state JOB_STATE_CANCELLED
ERROR:apache_beam.runners.dataflow.dataflow_runner:Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-05-12_08_51_56-17769454670639241342?project=<ProjectId>
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1675, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 10m 36s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/zks4et5eehyfc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #580
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/580/display/redirect?page=changes>
Changes:
[Alexey Romanenko] [BEAM-12918] Add PostCommit_Java_Tpcds_Spark job
[johnjcasey] [BEAM-14448] add datastore test
[yathu] [BEAM-14423] Add test cases for BigtableIO.BigtableWriterFn fails due to
[Pablo Estrada] Revert "Merge pull request #17517 from [BEAM-14383] Improve "FailedRows"
[noreply] [BEAM-14229] Fix SyntheticUnboundedSource duplication from checkpoint
[noreply] [BEAM-14347] Rename registration package to register (#17603)
[noreply] [BEAM-11104] Add self-checkpointing integration test (#17590)
------------------------------------------
[...truncated 329.79 KB...]
Using cached boto3-1.22.11-py3-none-any.whl (132 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting botocore<1.26.0,>=1.25.11
Using cached botocore-1.25.11-py3-none-any.whl (8.7 MB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.2.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2771396 sha256=a811fd0ff812424df545446cec059dcc115584fe513a58242e38eae19a54596b
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.9
Uninstalling pyparsing-3.0.9:
Successfully uninstalled pyparsing-3.0.9
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.12.0 boto3-1.22.11 botocore-1.25.11 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f21ade698c0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f21ade6a0e0> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0511100418.1652284424.766509/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0511100418.1652284424.766509/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0511100418.1652284424.766509/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0511100418.1652284424.766509/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220511155344767453-1661'
createTime: '2022-05-11T15:53:45.811472Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-11_08_53_45-2872587195487053483'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0511100418'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-11T15:53:45.811472Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-11_08_53_45-2872587195487053483]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-11_08_53_45-2872587195487053483
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-11_08_53_45-2872587195487053483?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-11_08_53_45-2872587195487053483 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:47.720Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.298Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.327Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.355Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.376Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.407Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.431Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.452Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.474Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.499Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.518Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.544Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.568Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.585Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.608Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.666Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.770Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.820Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.849Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:53:48.868Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-11_08_53_45-2872587195487053483 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:54:11.133Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:54:11.150Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:54:18.790Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:54:21.337Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T15:54:43.753Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T16:00:03Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T16:00:03.344Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T16:00:03.683Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T16:00:03.989Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T16:00:04.112Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T16:00:04.190Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T16:00:04.274Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T20:00:40.067Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-11_08_53_45-2872587195487053483.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T20:00:40.093Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T20:00:40.339Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T20:00:40.412Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T20:00:40.443Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-11_08_53_45-2872587195487053483 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T20:01:14.243Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T20:01:14.285Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-11T20:01:14.306Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-11_08_53_45-2872587195487053483 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1670, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 8m 46s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/de5pu6oci65nm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #579
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/579/display/redirect?page=changes>
Changes:
[andyye333] Add extra details to PubSub matcher errors
[noreply] Merge pull request #17559 from [BEAM-14423] Add exception injection
[noreply] [BEAM-11104] Allow self-checkpointing SDFs to return without finishing
[noreply] Merge pull request #17544 from [BEAM-14415] Exception handling tests for
[noreply] Merge pull request #17565 from [BEAM-14413] add Kafka exception test
[noreply] Merge pull request #17555 from [BEAM-14417] Adding exception handling
[noreply] [BEAM-14433] Improve Go split error message. (#17575)
[noreply] [BEAM-14429] Force java load test on dataflow runner v2
[noreply] Merge pull request #17577 from [BEAM-14435] Adding exception handling
[noreply] [BEAM-14347] Add generic registration functions for iters and emitters
[noreply] [BEAM-14169] Add Credentials rotation cron job for clusters (#17383)
[noreply] [BEAM-14347] Add generic registration for accumulators (#17579)
------------------------------------------
[...truncated 329.37 KB...]
Using cached google_cloud_vision-1.0.1-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-recommendations-ai<=0.2.0,>=0.1.0
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting botocore<1.26.0,>=1.25.10
Using cached botocore-1.25.10-py3-none-any.whl (8.7 MB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.2.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2771942 sha256=9d98bd1bdd8c9f2cf23184fbf3603d4b8129dc9383aa2e6b2875f113d9b03e2d
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.8
Uninstalling pyparsing-3.0.8:
Successfully uninstalled pyparsing-3.0.8
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.11.0 boto3-1.22.10 botocore-1.25.10 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f3b1511f170> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f3b1511f950> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0510100444.1652197663.217501/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0510100444.1652197663.217501/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0510100444.1652197663.217501/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0510100444.1652197663.217501/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220510154743218778-1850'
createTime: '2022-05-10T15:47:44.339849Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-10_08_47_43-12573554164792080462'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0510100444'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-10T15:47:44.339849Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-10_08_47_43-12573554164792080462]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-10_08_47_43-12573554164792080462
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-10_08_47_43-12573554164792080462?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-10_08_47_43-12573554164792080462 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:46.969Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.111Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.151Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.215Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.241Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.283Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.317Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.351Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.382Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.415Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.437Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.480Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.517Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.566Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.597Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.629Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.763Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.841Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.918Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:47:48.949Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-10_08_47_43-12573554164792080462 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:48:03.508Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:48:11.265Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:48:43.518Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:55:23.943Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:55:24.013Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:55:24.045Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:55:24.108Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:55:24.175Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:55:24.246Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T15:55:24.338Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T20:00:43.145Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-10_08_47_43-12573554164792080462.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T20:00:43.174Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T20:00:43.333Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T20:00:43.411Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T20:00:43.435Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-10_08_47_43-12573554164792080462 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T20:01:24.760Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T20:01:24.809Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-10T20:01:24.838Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-10_08_47_43-12573554164792080462 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1670, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 15m 2s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/23vfqyicyqq6s
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #578
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/578/display/redirect?page=changes>
Changes:
[elias.segundo] Changing elegibility to AllNodeElegibility
[chamikaramj] Adds code reviewers for GCP I/O connectors and KafkaIO to Beam OWNERS
------------------------------------------
[...truncated 328.59 KB...]
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting botocore<1.26.0,>=1.25.9
Using cached botocore-1.25.9-py3-none-any.whl (8.7 MB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.1.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2770580 sha256=a5ea0e2d61265da097a0eb73d0a21aedffdf4ec59c954dcb314e5c93ba980a70
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.8
Uninstalling pyparsing-3.0.8:
Successfully uninstalled pyparsing-3.0.8
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.11.0 boto3-1.22.9 botocore-1.25.9 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fd0b9af6200> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fd0b9af69e0> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0509100417.1652111232.656234/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0509100417.1652111232.656234/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0509100417.1652111232.656234/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0509100417.1652111232.656234/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220509154712657166-5456'
createTime: '2022-05-09T15:47:13.629691Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-09_08_47_13-12598048265630042825'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0509100417'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-09T15:47:13.629691Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-09_08_47_13-12598048265630042825]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-09_08_47_13-12598048265630042825
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-09_08_47_13-12598048265630042825?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-09_08_47_13-12598048265630042825 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:17.891Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.243Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.273Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.300Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.392Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.426Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.452Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.506Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.534Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.560Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.586Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.608Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.639Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.663Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.697Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.730Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.909Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:19.964Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:20.018Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:20.051Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-09_08_47_13-12598048265630042825 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:42.245Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 8 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:42.279Z: JOB_MESSAGE_DETAILED: Resized **** pool to 8, though goal was 10. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:42.482Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:47:52.462Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:48:14.753Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:53:50.018Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:53:50.110Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:53:50.136Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:53:50.179Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:53:50.276Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:53:50.336Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T15:53:50.394Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-09_08_47_13-12598048265630042825 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T20:00:40.320Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-09_08_47_13-12598048265630042825.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T20:00:40.344Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T20:00:43.567Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T20:00:43.779Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T20:00:43.806Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T20:01:17.193Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T20:01:17.231Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-09T20:01:17.253Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-09_08_47_13-12598048265630042825 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1670, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 15m 18s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/ejvzoozskvdb6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #577
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/577/display/redirect>
Changes:
------------------------------------------
[...truncated 328.96 KB...]
Using cached cryptography-37.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting botocore<1.26.0,>=1.25.9
Using cached botocore-1.25.9-py3-none-any.whl (8.7 MB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.1.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2769970 sha256=64aa88b216bbbd319c758cc7e081a2c3500d01185ce4e29438961b84dfa667e6
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.8
Uninstalling pyparsing-3.0.8:
Successfully uninstalled pyparsing-3.0.8
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.11.0 boto3-1.22.9 botocore-1.25.9 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f618ea0e200> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f618ea0e9e0> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0508100427.1652024924.679018/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0508100427.1652024924.679018/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0508100427.1652024924.679018/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0508100427.1652024924.679018/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220508154844679952-6931'
createTime: '2022-05-08T15:48:45.727694Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-08_08_48_45-5825726509403692619'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0508100427'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-08T15:48:45.727694Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-08_08_48_45-5825726509403692619]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-08_08_48_45-5825726509403692619
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-08_08_48_45-5825726509403692619?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-08_08_48_45-5825726509403692619 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:48.610Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:50.822Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:50.862Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:50.906Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:50.942Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.015Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.041Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.075Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.109Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.142Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.168Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.201Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.240Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.272Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.307Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.341Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.470Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.539Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.585Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:48:51.657Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-08_08_48_45-5825726509403692619 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:49:14.184Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:49:14.216Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:49:21.302Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:49:24.421Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:49:47.990Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:56:02.492Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:56:02.560Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:56:02.593Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:56:02.660Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:56:02.712Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:56:02.772Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T15:56:02.848Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T20:00:37.873Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-08_08_48_45-5825726509403692619.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T20:00:37.902Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T20:00:38.707Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T20:00:38.752Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T20:00:38.811Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-08_08_48_45-5825726509403692619 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T20:01:10.700Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T20:01:10.745Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-08T20:01:10.770Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-08_08_48_45-5825726509403692619 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1670, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 13m 37s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/tsgbb226drwgc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #576
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/576/display/redirect?page=changes>
Changes:
[kevinsijo] Setting up a basic directory
[kevinsijo] Mirroring Python SDK's directory structure
[kerrydc] Adds initial tests
[kevinsijo] 'runners' is the correct directory name
[Pablo Estrada] sketching the core API for JS SDK
[jonathanlui] add .gitignore for node/ts project
[Robert Bradshaw] Worker directory.
[Robert Bradshaw] Fix complile errors with explicit any for callables.
[Robert Bradshaw] Add worker entry point.
[Robert Bradshaw] Add proto generation code.
[Robert Bradshaw] Add generated proto files.
[Robert Bradshaw] Attempts to get ts protos to compile.
[Robert Bradshaw] Exclude ts protos for now.
[Robert Bradshaw] More changes to get ts protos working.
[Robert Bradshaw] Update scripts and config to get protos compiling.
[Robert Bradshaw] Update geenrated files.
[jonathanlui] add build and clean script to compile ts
[Robert Bradshaw] Generate server for loopback worker.
[Robert Bradshaw] Generated grpc servers for loopback.
[Robert Bradshaw] Add typescript formatter.
[Robert Bradshaw] Loopback server (that does nothing).
[Robert Bradshaw] Working server.
[Pablo Estrada] Starting expansion of primitive transforms
[Pablo Estrada] Starting to implement and support standard coders
[Robert Bradshaw] Also generate grpc clients.
[Robert Bradshaw] Basic implementation of worker harness.
[Pablo Estrada] fix the build
[Robert Bradshaw] Add some missing files for worker harness.
[Robert Bradshaw] Refactor operators to use registration.
[jonathanlui] enable ts in mocha
[jonathanlui] update readme
[jonathanlui] --save-dev @types/mocha
[jonathanlui] translate core_test.js to typescript
[Robert Bradshaw] Encapsulate worker service in a class.
[Kenneth Knowles] Port standard_coders_test to typescript (superficially)
[Pablo Estrada] Starting the proto translation of Impulse, ParDo, GBK
[Robert Bradshaw] Add some tests for the worker code.
[Robert Bradshaw] Fixing old lock file error.
[Pablo Estrada] Adding transform names and fixing GBK coder issue
[Robert Bradshaw] npx tsfmt -r src/apache_beam/base.ts src/apache_beam/transforms/core.ts
[Kenneth Knowles] switch to import style require() statements
[Kenneth Knowles] Add Coder interface using protobufjs classes
[Kenneth Knowles] BytesCoder with some failures
[noreply] Added GeneralObjectCoder and using it as coder for most transforms (#9)
[Kenneth Knowles] Fix order of arguments to deepEqual
[Kenneth Knowles] Encode expected encoding as binary
[Robert Bradshaw] Refactor API to allow for composites.
[jrmccluskey] Initial setup for automated Java expansion startup
[jrmccluskey] Update exp_service.ts
[Kenneth Knowles] Fix up coder deserialization
[Robert Bradshaw] Simplify GBK coder computation.
[Robert Bradshaw] Remove top-level PValue.
[Pablo Estrada] Make tests green
[Robert Bradshaw] Rename PValueish to PValue.
[jonathanlui] node runner
[jonathanlui] whitespaces
[Robert Bradshaw] Make Runner.run async.
[jonathanlui] bson and fast-deep-equal should not be listed as devdependency
[jrmccluskey] Add basic Dockerfile that starts ExternalWorkerPool
[Robert Bradshaw] Direct runner.
[kevinsijo] Testing expansion service communication
[Robert Bradshaw] Added flatten, assertion checkers.
[Pablo Estrada] progress on basic coders
[Robert Bradshaw] Fixing the build.
[Robert Bradshaw] Cleanup, simplify access.
[Pablo Estrada] Adding limited support for KVCoder and IterableCoder
[Robert Bradshaw] Introduce PipelineContext.
[Robert Bradshaw] Add toProto to all coders.
[Robert Bradshaw] Some work with coders.
[Robert Bradshaw] Remove debug logging.
[Robert Bradshaw] Use coders over data channel.
[Kenneth Knowles] explicitly sequence sub-coder serializations
[Kenneth Knowles] no more need to extend FakeCoder
[Kenneth Knowles] actually advance reader
[Kenneth Knowles] autoformat
[Kenneth Knowles] protobufjs already can write and read signed varints
[Kenneth Knowles] with improved test harness, kv has many more failures
[Kenneth Knowles] read bytescoder from correct position
[Kenneth Knowles] no more fake coders
[Kenneth Knowles] varint examples all work
[Kenneth Knowles] simplify coder value parsing
[Kenneth Knowles] global window coder
[Kenneth Knowles] fix swapEndian32
[Robert Bradshaw] Add P(...) operator.
[kevinsijo] Implementing RowCoder encoding.
[jrmccluskey] remove unused container dir
[kevinsijo] Corrected sorting of encoded positions to reflect an argsort instead.
[Robert Bradshaw] Populate environments.
[kevinsijo] Implementing RowCoder decoding.
[Kenneth Knowles] preliminary unbounded iterable coder
[Kenneth Knowles] friendlier description of standard coder test case
[Kenneth Knowles] fix test harness; iterable works
[jrmccluskey] first pass at boot.go
[jonathanlui] update package-lock.json
[jonathanlui] make NodeRunner a subclass of Runner
[jonathanlui] add waitUntilFinish interface member
[Pablo Estrada] Adding double coder
[Kenneth Knowles] scaffolding for windowed values
[Pablo Estrada] Adding type information to PColleciton and PTransform
[jonathanlui] fix direct runner
[Pablo Estrada] Adding typing information for DoFns
[Kenneth Knowles] add interval window
[Robert Bradshaw] Export PValue.
[Robert Bradshaw] Add CombineFn interface.
[Robert Bradshaw] Typed flatten.
[jonathanlui] add runAsync method to base.Runner
[Kenneth Knowles] add Long package
[Pablo Estrada] Adding more types. Making PValue typed
[Kenneth Knowles] instant coder draft
[Robert Bradshaw] Return job state from direct runner.
[Kenneth Knowles] type instant = long
[jonathanlui] implement NodeRunner.runPipeline
[Kenneth Knowles] autoformat
[kevinsijo] Completed implementation of basic row coder
[Kenneth Knowles] Fix IntervalWindowCoder, almost
[Kenneth Knowles] fix interval window coder
[Kenneth Knowles] autoformat
[Robert Bradshaw] loopback runner works
[Kenneth Knowles] move core element types into values.ts
[Kenneth Knowles] just build object directly to be cool
[Robert Bradshaw] GBK working on ULR.
[Robert Bradshaw] Async transforms.
[Robert Bradshaw] External transform grpah splicing.
[Kenneth Knowles] progress on windowed value: paneinfo encoding
[Robert Bradshaw] Fix merge.
[Robert Bradshaw] autoformat
[Kenneth Knowles] full windowed value coder
[kerrydc] Updates tests to use correct types, adds generics where needed to DoFns
[Robert Bradshaw] Add serialization librarires.'
[Robert Bradshaw] Add Split() PTransform, for producing multiple outputs from a single
[Robert Bradshaw] Schema-encoded external payloads.
[kevinsijo] Adding Schema inference from JSON
[Pablo Estrada] Removing unused directories
[Pablo Estrada] Support for finishBundle and improving typing annotations.
[Pablo Estrada] A base implementation of combiners with GBK/ParDo
[Robert Bradshaw] Fully propagate windowing information in both remote and direct runner.
[Robert Bradshaw] Make args and kwargs optional for python external transform.
[Robert Bradshaw] Infer schema for external transforms.
[Pablo Estrada] Implementing a custom combine fn as an example. Small fixes
[Robert Bradshaw] Fix missing windowing information in combiners.
[Robert Bradshaw] PostShuffle needn't group by key as that's already done.
[Robert Bradshaw] Guard pre-combine for global window only.
[Robert Bradshaw] WindowInto
[Robert Bradshaw] Fix optional kwargs.
[Robert Bradshaw] A couple of tweaks for js + py
[Robert Bradshaw] Add windowing file.
[Robert Bradshaw] CombineBy transform, stand-alone WordCount.
[Robert Bradshaw] cleanup
[Robert Bradshaw] Actually fix optional external kwargs.
[Robert Bradshaw] Demo2, textio read.
[Robert Bradshaw] Add command lines for starting up the servers.
[Robert Bradshaw] Run prettier on the full codebase.
[Robert Bradshaw] Update deps.
[Pablo Estrada] Adding docstrings for core.ts. Prettier dependency
[Pablo Estrada] Documenting coder interfaces
[Pablo Estrada] Added documentation for a few standard coders
[Robert Bradshaw] Unified grouping and combining.
[Robert Bradshaw] Allow PCollection ids to be lazy.
[Robert Bradshaw] Reorganize module structure.
[Robert Bradshaw] A couple more renames.
[Robert Bradshaw] Simplify.
[Robert Bradshaw] Consolidation.
[Robert Bradshaw] Fix build.
[Robert Bradshaw] Add optional context to ParDo.
[Robert Bradshaw] fixup: iterable coder endian sign issue
[Robert Bradshaw] omit context for map(console.log)
[Robert Bradshaw] Fix ReadFromText coders.
[Robert Bradshaw] Flesh out README with overview and current state.
[noreply] Readme typo
[Robert Bradshaw] Two more TODOs.
[noreply] Add a pointer to the example wordcount to the readme.
[Pablo Estrada] Documenting coders and implementing unknown-length method
[Robert Bradshaw] UIID dependency.
[Robert Bradshaw] Artifact handling.
[Robert Bradshaw] Properly wait on data channel for bundle completion.
[Robert Bradshaw] Automatic java expansion service startup.
[Robert Bradshaw] Process promises.
[Robert Bradshaw] Implement side inputs.
[Robert Bradshaw] Cleanup.
[Robert Bradshaw] Put complex constext stuff in its own file.
[Robert Bradshaw] Rename BoundedWindow to just Window.
[Robert Bradshaw] Alternative splitter class.
[Pablo Estrada] Documenting internal functions
[Robert Bradshaw] Take a pass clarifying the TODOs.
[Robert Bradshaw] Sql transform wrapper.
[Robert Bradshaw] Incorporate some feedback into the TODOs.
[Robert Bradshaw] More TODOs.
[Robert Bradshaw] Remove app placeholder.
[Robert Bradshaw] Apache license headers.
[Robert Bradshaw] More TODOs
[jankuehle] Suggestions for TypeScript todos
[dannymccormick] Add actions for typescript sdk
[dannymccormick] Fix test command
[noreply] Add missing version
[dannymccormick] Fix codecovTest command
[noreply] Only do prettier check on linux
[noreply] Only get codecov on linux
[Robert Bradshaw] Resolve some comments.
[Robert Bradshaw] Fix compile errors.
[Robert Bradshaw] Prettier.
[Robert Bradshaw] Re-order expandInternal arguments pending unification.
[Robert Bradshaw] More consistent and stricter PTransform naming.
[Robert Bradshaw] Notes on explicit, if less idiomatic, use of classes.
[Robert Bradshaw] Let DoFn be an interface rather than a class.
[Robert Bradshaw] Provide DoFn context to start and finish bundle.
[Robert Bradshaw] Optional promise code simplification.
[Robert Bradshaw] Cleanup todos.
[Robert Bradshaw] Avoid any type where not needed.
[Robert Bradshaw] Apache RAT excludes for typescript.
[Robert Bradshaw] Remove empty READMEs.
[Robert Bradshaw] Add licences statement to readme files.
[Robert Bradshaw] More RAT fixes.
[Robert Bradshaw] Another unsupported coder.
[Robert Bradshaw] Remove debugging code.
[noreply] Fix automatic naming with code coverage.
[Robert Bradshaw] Coders cleanup.
[Robert Bradshaw] Add tests for RowCoder.
[Robert Bradshaw] Normalize capitalization, comments.
[Robert Bradshaw] Install typescript closure packages.
[Robert Bradshaw] npm audit fix
[Robert Bradshaw] Move more imports out of base.
[Robert Bradshaw] Changes needed to compile with ts closure plugin.
[Robert Bradshaw] Use ttsc and ts-closure-transform plugin.
[Robert Bradshaw] Serialization registration to actually get serialization working.
[Robert Bradshaw] Container images working on local runner.
[Robert Bradshaw] Add a portable job server that proxies the Dataflow backend. (#17189)
[Robert Bradshaw] Improvements to dataflow job service for non-Python jobs.
[Robert Bradshaw] Get dataflow working.
[Robert Bradshaw] User friendly pipeline options.
[Robert Bradshaw] Less classes, more functions.
[Robert Bradshaw] Add new nullable standard coder.
[Robert Bradshaw] Make Apache Rat happy.
[Robert Bradshaw] Disable broken codecov.
[Robert Bradshaw] Remove last uses of base.ts.
[Robert Bradshaw] Remove unneedd file.
[Robert Bradshaw] Remove more uneeded/unused files.
[Robert Bradshaw] Cleanup tests.
[Robert Bradshaw] Minor cleanups to coder tests.
[noreply] Quote pip install package name
[noreply] [BEAM-14374] Fix module import error in FullyQualifiedNamedTransform
[Robert Bradshaw] Addressing issues from the review.
[noreply] Apply suggestions from code review.
[Robert Bradshaw] Post-merge fixes.
[dannymccormick] Delete tags.go
[Robert Bradshaw] Update tests to use our actual serialization libraries.
[Robert Bradshaw] Another pass at TODOs, removing finished items.
[Heejong Lee] [BEAM-14146] Python Streaming job failing to drain with BigQueryIO write
[Kenneth Knowles] Add parameter for service account impersonation in GCP credentials
[Heejong Lee] add test
[noreply] Merge pull request #17490 from [BEAM-14370] [Website] Add new page about
[noreply] [BEAM-14332] Refactored cluster management for Flink on Dataproc
[noreply] [BEAM-13988] Update mtime to use time.UnixMilli() calls (#17578)
[noreply] Fixing patching error on missing dependencies (#17564)
[noreply] Merge pull request #17517 from [BEAM-14383] Improve "FailedRows" errors
[Heejong Lee] add test without mock
------------------------------------------
[...truncated 329.20 KB...]
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Collecting boto3>=1.9
Using cached boto3-1.22.9-py3-none-any.whl (132 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting botocore<1.26.0,>=1.25.9
Using cached botocore-1.25.9-py3-none-any.whl (8.7 MB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.1.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2769970 sha256=9cc9be49ba12636d6a0921516bc994bcbdb6a88e605ae2233ee2e01e4f136ec8
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.8
Uninstalling pyparsing-3.0.8:
Successfully uninstalled pyparsing-3.0.8
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.24.0 azure-storage-blob-12.11.0 boto3-1.22.9 botocore-1.25.9 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fcf2d35f200> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fcf2d35f9e0> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0507100427.1651938352.120670/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0507100427.1651938352.120670/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0507100427.1651938352.120670/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0507100427.1651938352.120670/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220507154552121646-4234'
createTime: '2022-05-07T15:45:53.140011Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-07_08_45_52-13998133551949395336'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0507100427'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-07T15:45:53.140011Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-07_08_45_52-13998133551949395336]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-07_08_45_52-13998133551949395336
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-07_08_45_52-13998133551949395336?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-07_08_45_52-13998133551949395336 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:54.953Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:55.704Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:55.813Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:55.864Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:55.902Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:55.965Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.027Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.052Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.090Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.122Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.157Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.186Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.224Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.257Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.284Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.309Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.470Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.532Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.579Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:45:56.601Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-07_08_45_52-13998133551949395336 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:46:19.012Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:46:27.154Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:46:51.387Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:53:05.452Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:53:05.553Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:53:05.586Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:53:05.673Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:53:05.730Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:53:05.798Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T15:53:05.885Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T20:00:41.823Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-07_08_45_52-13998133551949395336.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T20:00:41.852Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T20:00:41.960Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T20:00:42.101Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T20:00:42.124Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-07_08_45_52-13998133551949395336 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T20:01:16.423Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T20:01:16.458Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-07T20:01:16.485Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-07_08_45_52-13998133551949395336 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1670, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 16m 38s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/ghq57sqyqphtc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #575
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/575/display/redirect?page=changes>
Changes:
[zyichi] Move master readme.md to 2.40.0
[noreply] [BEAM-14173] Fix Go Loadtests on Dataflow & partial fix for Flink
[noreply] Upgrade python sdk container requirements. (#17549)
[noreply] Merge pull request #17497: [BEAM-11205] Update GCP Libraries BOM version
[noreply] [BEAM-12603] Add retry on grpc data channel and remove retry from test.
[noreply] Merge pull request #17359: [BEAM-14303] Add a way to exclude output
[noreply] [BEAM-14347] Allow users to optimize DoFn execution with a single
[noreply] [BEAM-5878] Add (failing) kwonly-argument test (#17509)
------------------------------------------
[...truncated 329.76 KB...]
Using cached testcontainers-3.5.3-py2.py3-none-any.whl (35 kB)
Collecting cryptography>=36.0.0
Using cached cryptography-37.0.2-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting botocore<1.26.0,>=1.25.8
Using cached botocore-1.25.8-py3-none-any.whl (8.7 MB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.1.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.13.0-py3-none-any.whl (51 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2766904 sha256=04a6b17ed4a5982a4646e9d864711cb74f02adad69e619eef684e6fc68bc940a
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.8
Uninstalling pyparsing-3.0.8:
Successfully uninstalled pyparsing-3.0.8
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.23.1 azure-storage-blob-12.11.0 boto3-1.22.8 botocore-1.25.8 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.13.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fb08e6f4170> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fb08e6f4950> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0506100420.1651852937.003077/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0506100420.1651852937.003077/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0506100420.1651852937.003077/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0506100420.1651852937.003077/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220506160217004048-5230'
createTime: '2022-05-06T16:02:18.292725Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-06_09_02_17-2928937170520540250'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0506100420'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-06T16:02:18.292725Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-06_09_02_17-2928937170520540250]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-06_09_02_17-2928937170520540250
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-06_09_02_17-2928937170520540250?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-06_09_02_17-2928937170520540250 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:21.154Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.658Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.733Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.769Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.800Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.846Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.874Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.906Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.941Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.966Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:22.988Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:23.011Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:23.057Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:23.096Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:23.126Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:23.160Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:23.320Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:23.384Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-06_09_02_17-2928937170520540250 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:23.428Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:23.480Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:46.469Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:02:50.771Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:03:18.592Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:09:58.291Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:09:58.403Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:09:58.445Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:09:58.560Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:09:58.616Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:09:58.693Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T16:09:58.764Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T20:00:39.344Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-06_09_02_17-2928937170520540250.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T20:00:39.377Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T20:00:39.579Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T20:00:39.720Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T20:00:39.766Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-06_09_02_17-2928937170520540250 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T20:01:14.273Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T20:01:14.315Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-06T20:01:14.343Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-06_09_02_17-2928937170520540250 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1670, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 18s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/jxaemmykxxshg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #574
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/574/display/redirect?page=changes>
Changes:
[Heejong Lee] [BEAM-9245] Unable to pull datatore Entity which contains dict
[bulat.safiullin] [BEAM-14382] [Website] add banner container for with css, images, html
[Jan Lukavský] [BEAM-14196] add test verifying output watermark propagation in bundle
[Jan Lukavský] [BEAM-14196] Fix FlinkRunner mid-bundle output watermark handling
[nielm] [BEAM-14405] Fix NPE when ProjectID is not specified in a template
[bulat.safiullin] [BEAM-14382] change mobile banner img, add padding to banner section
[ahmedabualsaud] fix test decotrator typo
[noreply] Merge pull request #17440 from [BEAM-14329] Enable exponential backoff
[noreply] [BEAM-11104] Fix output forwarding issue for ProcessContinuations
[noreply] re-add testing package to pydoc (#17524)
[Heejong Lee] add test
[noreply] [BEAM-14250] Amended the workaround (#17531)
[noreply] [BEAM-11104] Fix broken split result validation (#17546)
[noreply] Fixed a SQL and screenshots in the Beam SQL blog (#17545)
[noreply] Merge pull request #17417: [BEAM-14388] Address some performance
[noreply] [BEAM-14386] [Flink] Support for scala 2.12 (#17512)
[noreply] [BEAM-14294] Worker changes to support trivial Batched DoFns (#17384)
[zyichi] Moving to 2.40.0-SNAPSHOT on master branch.
[noreply] [BEAM-14048] [CdapIO] Add ConfigWrapper for building CDAP PluginConfigs
------------------------------------------
[...truncated 330.18 KB...]
Using cached google_cloud_vision-1.0.1-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-recommendations-ai<=0.2.0,>=0.1.0
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.40.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting botocore<1.26.0,>=1.25.7
Using cached botocore-1.25.7-py3-none-any.whl (8.7 MB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.40.0.dev0) (21.3)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.40.0.dev0) (62.1.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.9.0-py2.py3-none-any.whl (112 kB)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.12.0-py3-none-any.whl (54 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (1.11.0)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (2.1.3)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.40.0.dev0) (3.8.0)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.40.0.dev0-py3-none-any.whl size=2766392 sha256=7118c32d29ab186ee10c2340b20c3af4f80ce921ab6befeff70864226f9d2194
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.8
Uninstalling pyparsing-3.0.8:
Successfully uninstalled pyparsing-3.0.8
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.40.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.23.1 azure-storage-blob-12.11.0 boto3-1.22.7 botocore-1.25.7 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.12.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.9.0 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.40.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fea6f2d8170> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fea6f2d8950> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0505100417.1651766455.004017/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0505100417.1651766455.004017/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0505100417.1651766455.004017/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0505100417.1651766455.004017/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220505160055004971-2689'
createTime: '2022-05-05T16:00:56.643556Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-05_09_00_55-11769499013114091023'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0505100417'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-05T16:00:56.643556Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-05_09_00_55-11769499013114091023]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-05_09_00_55-11769499013114091023
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-05_09_00_55-11769499013114091023?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-05_09_00_55-11769499013114091023 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:51.827Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:52.775Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:52.823Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:52.873Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:52.910Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:52.965Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.008Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.040Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.109Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.143Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.177Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.209Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.281Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.316Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.359Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.397Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.617Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.708Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.772Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:01:53.815Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-05_09_00_55-11769499013114091023 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:02:15.664Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:02:24.663Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:02:48.700Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:09:00.775Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:09:00.920Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:09:00.955Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:09:01.035Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:09:01.109Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:09:01.252Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T16:09:01.346Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T20:00:42.754Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-05_09_00_55-11769499013114091023.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T20:00:42.794Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T20:00:42.961Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-05_09_00_55-11769499013114091023 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T20:00:43.085Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T20:00:43.104Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T20:01:22.449Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T20:01:22.503Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-05T20:01:22.535Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-05_09_00_55-11769499013114091023 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1670, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 1m 44s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/7bciwxkxg5cb4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #573
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/573/display/redirect?page=changes>
Changes:
[noreply] fix: JDBC config schema fields order
[Brian Hulette] Revert "Merge pull request #17255 from kileys/test-revert"
[Brian Hulette] BEAM-14231: bypass schema cache for
[noreply] [BEAM-13657] Follow up update version warning in __init__ (#17493)
[noreply] Merge pull request #17431 from [BEAM-14273] Add integration tests for BQ
[noreply] Merge pull request #17205 from [BEAM-14145] [Website] add carousel to
[noreply] [BEAM-14064] fix es io windowing (#17112)
[noreply] [BEAM-13670] Upgraded ipython from v7 to v8 (#17529)
[noreply] [BEAM-11104] Enable ProcessContinuation return values, add unit test
[Robert Bradshaw] [BEAM-14403] Allow Prime to be used with legacy workers.
[noreply] [BEAM-11106] Support drain in Go SDK (#17432)
[noreply] add __Init__ to inference. (#17514)
------------------------------------------
[...truncated 328.90 KB...]
Using cached google_cloud_recommendations_ai-0.2.0-py2.py3-none-any.whl (180 kB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from azure-core>=1.7.0->apache-beam==2.39.0.dev0) (1.16.0)
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting botocore<1.26.0,>=1.25.6
Using cached botocore-1.25.6-py3-none-any.whl (8.7 MB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.39.0.dev0) (21.3)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.39.0.dev0) (62.1.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.46.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.8.1-py2.py3-none-any.whl (113 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.39.0.dev0) (1.11.0)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.39.0.dev0) (2.1.3)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.12.0-py3-none-any.whl (54 kB)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.39.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.39.0.dev0-py3-none-any.whl size=2759580 sha256=4f146c474b1a5f3c9667d1a13ab9fea9f96ba05cfeea289e845034a796cab769
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.8
Uninstalling pyparsing-3.0.8:
Successfully uninstalled pyparsing-3.0.8
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.39.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.23.1 azure-storage-blob-12.11.0 boto3-1.22.6 botocore-1.25.6 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.2 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.46.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.12.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.8.1 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.39.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7f2492833e60> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f2492838680> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651679455.910396/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651679455.910396/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651679455.910396/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651679455.910396/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220504155055911344-9566'
createTime: '2022-05-04T15:50:57.052430Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-04_08_50_56-13240567463436534960'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0428111049'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-04T15:50:57.052430Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-04_08_50_56-13240567463436534960]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-04_08_50_56-13240567463436534960
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-04_08_50_56-13240567463436534960?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-04_08_50_56-13240567463436534960 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:50:59.035Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-b.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:50:59.824Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:50:59.864Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:50:59.917Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:50:59.948Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:50:59.988Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.022Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.058Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.091Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.113Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.147Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.182Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.213Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.247Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.271Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.303Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.494Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.552Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.623Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:00.669Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-b...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-04_08_50_56-13240567463436534960 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:25.094Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:25.126Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:28.525Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:35.292Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:51:59.075Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:58:12.903Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:58:12.973Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:58:13.010Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:58:13.084Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:58:13.137Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:58:13.192Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T15:58:13.258Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T20:00:40.238Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-04_08_50_56-13240567463436534960.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T20:00:40.260Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T20:00:41.319Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T20:00:41.363Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T20:00:41.387Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-04_08_50_56-13240567463436534960 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T20:01:14.778Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T20:01:14.816Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-04T20:01:14.836Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-04_08_50_56-13240567463436534960 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1670, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 11m 44s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/kdodeimagdwru
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Python_SideInput_Dataflow_Batch #572
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/572/display/redirect?page=changes>
Changes:
[yathu] [BEAM-14375] Fix Java Wordcount Dataflow postcommit
[Robert Bradshaw] Allow arithmetic between deferred scalars.
[noreply] [BEAM-14390] Set user-agent when pulling licenses to avoid 403s (#17521)
[noreply] [BEAM-8688] Upgrade GCSIO to 2.2.6 (#17486)
[noreply] [BEAM-14253] patch SubscriptionPartitionLoader to work around a dataflow
[noreply] Add website link log to notify user of pre-build workflow. (#17498)
[noreply] [BEAM-11105] Add timestamp observing watermark estimation (#17476)
[noreply] Merge pull request #17487 from Adding user-agent to GCS client in Python
[noreply] [BEAM-10265] Display error message if trying to infer recursive schema
[noreply] [BEAM-12575] Upgraded ipykernel from v5 to v6 (#17526)
[noreply] [BEAM-11105] Add docs + CHANGES.md entry for Go Watermark Estimation
[noreply] Merge pull request #17380 from [BEAM-14314][BEAM-9532] Add last_updated
------------------------------------------
[...truncated 329.11 KB...]
Collecting msrest>=0.6.21
Using cached msrest-0.6.21-py2.py3-none-any.whl (85 kB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.0-py3-none-any.whl (23 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Using cached s3transfer-0.5.2-py3-none-any.whl (79 kB)
Collecting botocore<1.26.0,>=1.25.5
Using cached botocore-1.25.5-py3-none-any.whl (8.7 MB)
Collecting cffi>=1.12
Using cached cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.8-py3-none-any.whl (39 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Requirement already satisfied: packaging>=14.3 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-cloud-bigquery<3,>=1.6.0->apache-beam==2.39.0.dev0) (21.3)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.3-py3-none-any.whl (114 kB)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.3.2-py2.py3-none-any.whl (76 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting google-api-core<2.0.0dev,>=1.21.0
Using cached google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.18.0
Using cached google_auth-1.35.0-py2.py3-none-any.whl (152 kB)
Requirement already satisfied: setuptools>=40.3.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from google-auth<3,>=1.18.0->apache-beam==2.39.0.dev0) (62.1.0)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.44.0-py3-none-any.whl (10.0 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting pyparsing<3,>=2.4.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pbr>=0.11
Using cached pbr-5.8.1-py2.py3-none-any.whl (113 kB)
Requirement already satisfied: py>=1.5.0 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.39.0.dev0) (1.11.0)
Collecting more-itertools>=4.0.0
Using cached more_itertools-8.12.0-py3-none-any.whl (54 kB)
Collecting atomicwrites>=1.0
Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Requirement already satisfied: importlib-metadata>=0.12 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from pytest<5.0,>=4.4.0->apache-beam==2.39.0.dev0) (2.1.3)
Collecting pluggy<1.0,>=0.12
Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting wcwidth
Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting attrs>=17.4.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting scipy>=1.1.0
Using cached scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (150 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (75 kB)
Collecting docker
Using cached docker-5.0.3-py2.py3-none-any.whl (146 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
Using cached googleapis_common_protos-1.56.0-py2.py3-none-any.whl (241 kB)
Collecting google-api-core[grpc,grpcgcp]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.7.2-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.7.0-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.1-py3-none-any.whl (114 kB)
Using cached google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)
Using cached google_api_core-2.5.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)
Using cached google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38 kB)
Requirement already satisfied: zipp>=0.5 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from importlib-metadata>=0.12->pytest<5.0,>=4.4.0->apache-beam==2.39.0.dev0) (3.8.0)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting typing-utils>=0.0.3
Using cached typing_utils-0.1.0-py3-none-any.whl (10 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.3.2-py3-none-any.whl (54 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.0-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.39.0.dev0-py3-none-any.whl size=2740861 sha256=4b1efe05012cf2f198db8f61396e39c1f0f14ce125edf2f356a4f8558575ef74
Stored in directory: /home/jenkins/.cache/pip/wheels/c1/f5/a3/8f5ec48b796c1b690bd340ec7688fb88886afa83d7f14ff737
Successfully built apache-beam
Installing collected packages: wcwidth, pytz, pyasn1, parameterized, docopt, crcmod, certifi, wrapt, websocket-client, urllib3, typing-utils, typing-extensions, threadpoolctl, tenacity, rsa, pyyaml, python-dateutil, pyparsing, pymysql, pymongo, pyhamcrest, pycparser, pyasn1-modules, psycopg2-binary, proto-plus, pbr, orjson, oauthlib, numpy, more-itertools, joblib, jmespath, isodate, idna, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, dill, cloudpickle, charset-normalizer, cachetools, attrs, atomicwrites, sqlalchemy, scipy, requests, pydot, pyarrow, pluggy, pandas, overrides, mock, httplib2, grpcio-status, grpcio-gcp, google-resumable-media, google-auth, freezegun, cffi, botocore, scikit-learn, s3transfer, requests-oauthlib, requests_mock, pytest, oauth2client, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-api-core, docker, deprecation, cryptography, azure-core, testcontainers, pytest-timeout, pytest-forked, msrest, google-cloud-core, google-apitools, boto3, apache-beam, pytest-xdist, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, google-cloud-bigquery, azure-storage-blob, google-cloud-pubsublite
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.8
Uninstalling pyparsing-3.0.8:
Successfully uninstalled pyparsing-3.0.8
Attempting uninstall: pluggy
Found existing installation: pluggy 1.0.0
Uninstalling pluggy-1.0.0:
Successfully uninstalled pluggy-1.0.0
Successfully installed apache-beam-2.39.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.23.1 azure-storage-blob-12.11.0 boto3-1.22.5 botocore-1.25.5 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.12 cloudpickle-2.0.0 crcmod-1.7 cryptography-37.0.1 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.11 fasteners-0.17.3 freezegun-1.2.1 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-2.34.3 google-cloud-bigquery-storage-2.13.1 google-cloud-bigtable-1.7.1 google-cloud-core-1.7.2 google-cloud-datastore-1.15.4 google-cloud-dlp-3.6.2 google-cloud-language-1.3.1 google-cloud-pubsub-2.12.0 google-cloud-pubsublite-1.4.2 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.2 google-cloud-videointelligence-1.16.2 google-cloud-vision-1.0.1 google-crc32c-1.3.0 google-resumable-media-2.3.2 googleapis-common-protos-1.56.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.4 grpcio-gcp-0.2.2 grpcio-status-1.44.0 hdfs-2.7.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-1.0.0 joblib-1.1.0 mock-2.0.0 more-itertools-8.12.0 msrest-0.6.21 numpy-1.21.6 oauth2client-4.1.3 oauthlib-3.2.0 orjson-3.6.8 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.8.1 pluggy-0.13.1 proto-plus-1.20.3 psycopg2-binary-2.9.3 pyarrow-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2022.1 pyyaml-6.0 requests-2.27.1 requests-oauthlib-1.3.1 requests_mock-1.9.3 rsa-4.8 s3transfer-0.5.2 scikit-learn-1.0.2 scipy-1.7.3 sqlalchemy-1.4.36 tenacity-5.1.5 testcontainers-3.5.3 threadpoolctl-3.1.0 typing-extensions-4.2.0 typing-utils-0.1.0 urllib3-1.26.9 wcwidth-0.2.5 websocket-client-1.3.2 wrapt-1.14.1
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:root:100 out of 10000 total elements in each window will be accessed.
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:root:Default Python SDK image for environment is apache/beam_python3.7_sdk:2.39.0.dev
INFO:root:Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407
INFO:root:Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python37:beam-master-20220407" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x7fd8db172dd0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7fd8db1735f0> ====================
INFO:apache_beam.runners.dataflow.internal.apiclient:Defaulting to the temp_location as staging_location: gs://temp-storage-for-perf-tests/loadtests
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60 seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651591758.946113/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651591758.946113/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651591758.946113/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-sideinput-9-0428111049.1651591758.946113/pipeline.pb in 0 seconds.
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['shuffle_mode=appliance', '--side_input_type=iter', '--access_percentage=1', '--window_count=1000']
INFO:apache_beam.runners.dataflow.internal.apiclient:Create job: <Job
clientRequestId: '20220503152918947038-5293'
createTime: '2022-05-03T15:29:19.926456Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-05-03_08_29_19-8254765706794469445'
location: 'us-central1'
name: 'load-tests-python-dataflow-batch-sideinput-9-0428111049'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-05-03T15:29:19.926456Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO:apache_beam.runners.dataflow.internal.apiclient:Created job with id: [2022-05-03_08_29_19-8254765706794469445]
INFO:apache_beam.runners.dataflow.internal.apiclient:Submitted job: 2022-05-03_08_29_19-8254765706794469445
INFO:apache_beam.runners.dataflow.internal.apiclient:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-03_08_29_19-8254765706794469445?project=apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-03_08_29_19-8254765706794469445 is in state JOB_STATE_PENDING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:23.349Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-f.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.172Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.215Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.245Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.274Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.305Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.344Z: JOB_MESSAGE_DETAILED: Fusing consumer Get synthetic SDF options into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.382Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect start time metrics into Apply windows
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.420Z: JOB_MESSAGE_DETAILED: Fusing consumer Assign timestamps into Create/Read
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.494Z: JOB_MESSAGE_DETAILED: Fusing consumer Apply windows into Assign timestamps
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.549Z: JOB_MESSAGE_DETAILED: Fusing consumer Generate input into Get synthetic SDF options
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.579Z: JOB_MESSAGE_DETAILED: Fusing consumer Collect end time metrics into ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.624Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.656Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.698Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.736Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.879Z: JOB_MESSAGE_DEBUG: Executing wait step start4
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:24.936Z: JOB_MESSAGE_BASIC: Executing operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:25.004Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-03_08_29_19-8254765706794469445 is in state JOB_STATE_RUNNING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:25.036Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:48.084Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:29:55.927Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:30:20.712Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:36:05.495Z: JOB_MESSAGE_BASIC: Finished operation Create/Read+Assign timestamps+Apply windows+Get synthetic SDF options+Collect start time metrics+Generate input
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:36:05.577Z: JOB_MESSAGE_DEBUG: Value "Collect start time metrics.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:36:05.613Z: JOB_MESSAGE_DEBUG: Value "Generate input.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:36:05.674Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:36:05.735Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:36:05.796Z: JOB_MESSAGE_DEBUG: Value "ParDo(SequenceSideInputTestDoFn)/_UnpickledSideInput(Generate input.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T15:36:05.860Z: JOB_MESSAGE_BASIC: Executing operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
Daemon will be stopped at the end of the build after the daemon was no longer found in the daemon registry
> Task :sdks:python:apache_beam:testing:load_tests:run
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T20:00:40.013Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2022-05-03_08_29_19-8254765706794469445.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T20:00:40.052Z: JOB_MESSAGE_BASIC: Finished operation ParDo(SequenceSideInputTestDoFn)/ParDo(SequenceSideInputTestDoFn)+Collect end time metrics
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T20:00:40.246Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T20:00:40.351Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T20:00:40.404Z: JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-03_08_29_19-8254765706794469445 is in state JOB_STATE_CANCELLING
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T20:01:14.886Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T20:01:14.958Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2022-05-03T20:01:14.986Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-05-03_08_29_19-8254765706794469445 is in state JOB_STATE_CANCELLED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/sideinput_test.py",> line 216, in <module>
SideInputTest().run()
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py",> line 151, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File "<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1670, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: CANCELLED, Error:
None
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Python_SideInput_Dataflow_Batch/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'> line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 4h 33m 8s
15 actionable tasks: 3 executed, 12 up-to-date
Publishing build scan...
https://gradle.com/s/eaun622rx5iye
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org