You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2022/10/16 13:06:41 UTC
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1005
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1005/display/redirect>
Changes:
------------------------------------------
[...truncated 20.75 KB...]
Using cached pytest-7.1.3-py3-none-any.whl (298 kB)
Collecting pytest-xdist<3,>=2.5.0
Using cached pytest_xdist-2.5.0-py3-none-any.whl (41 kB)
Collecting pytest-timeout<3,>=2.1.0
Using cached pytest_timeout-2.1.0-py3-none-any.whl (12 kB)
Collecting scikit-learn>=0.20.0
Using cached scikit_learn-1.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30.8 MB)
Collecting sqlalchemy<2.0,>=1.3
Using cached SQLAlchemy-1.4.41-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Collecting psycopg2-binary<3.0.0,>=2.8.5
Using cached psycopg2_binary-2.9.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
Collecting testcontainers[mysql]<4.0.0,>=3.0.3
Using cached testcontainers-3.7.0-py2.py3-none-any.whl (43 kB)
Collecting cryptography>=36.0.0
Using cached cryptography-38.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (4.2 MB)
Collecting cachetools<5,>=3.1.0
Using cached cachetools-4.2.4-py3-none-any.whl (10 kB)
Collecting google-apitools<0.5.32,>=0.5.31
Using cached google_apitools-0.5.31-py3-none-any.whl
Collecting google-auth<3,>=1.18.0
Using cached google_auth-2.12.0-py2.py3-none-any.whl (169 kB)
Collecting google-auth-httplib2<0.2.0,>=0.1.0
Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting google-cloud-datastore<2,>=1.8.0
Using cached google_cloud_datastore-1.15.5-py2.py3-none-any.whl (134 kB)
Collecting google-cloud-pubsub<3,>=2.1.0
Using cached google_cloud_pubsub-2.13.10-py2.py3-none-any.whl (236 kB)
Collecting google-cloud-pubsublite<2,>=1.2.0
Using cached google_cloud_pubsublite-1.5.0-py2.py3-none-any.whl (270 kB)
Collecting google-cloud-bigquery<4,>=1.6.0
Using cached google_cloud_bigquery-3.3.5-py2.py3-none-any.whl (211 kB)
Collecting google-cloud-bigquery-storage<2.14,>=2.6.3
Using cached google_cloud_bigquery_storage-2.13.2-py2.py3-none-any.whl (180 kB)
Collecting google-cloud-core<3,>=0.28.1
Using cached google_cloud_core-2.3.2-py2.py3-none-any.whl (29 kB)
Collecting google-cloud-bigtable<2,>=0.31.1
Using cached google_cloud_bigtable-1.7.2-py2.py3-none-any.whl (267 kB)
Collecting google-cloud-spanner<4,>=3.0.0
Using cached google_cloud_spanner-3.22.2-py2.py3-none-any.whl (291 kB)
Collecting google-cloud-dlp<4,>=3.0.0
Using cached google_cloud_dlp-3.9.2-py2.py3-none-any.whl (125 kB)
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.2-py2.py3-none-any.whl (83 kB)
Collecting google-cloud-videointelligence<2,>=1.8.0
Using cached google_cloud_videointelligence-1.16.3-py2.py3-none-any.whl (183 kB)
Collecting google-cloud-vision<2,>=0.38.0
Using cached google_cloud_vision-1.0.2-py2.py3-none-any.whl (435 kB)
Collecting google-cloud-recommendations-ai<0.8.0,>=0.1.0
Using cached google_cloud_recommendations_ai-0.7.1-py2.py3-none-any.whl (148 kB)
Collecting pandas<2.0.0
Using cached pandas-1.4.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
Requirement already satisfied: six>=1.11.0 in <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages> (from azure-core>=1.7.0->apache-beam==2.43.0.dev0) (1.16.0)
Collecting msrest>=0.7.1
Using cached msrest-0.7.1-py3-none-any.whl (85 kB)
Collecting s3transfer<0.7.0,>=0.6.0
Using cached s3transfer-0.6.0-py3-none-any.whl (79 kB)
Collecting botocore<1.28.0,>=1.27.91
Using cached botocore-1.27.91-py3-none-any.whl (9.2 MB)
Collecting jmespath<2.0.0,>=0.7.1
Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
Collecting oauth2client>=1.4.12
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.9-py3-none-any.whl (34 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
Using cached google_api_core-2.10.2-py3-none-any.whl (115 kB)
Requirement already satisfied: packaging<22.0.0dev,>=14.3 in <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages> (from google-cloud-bigquery<4,>=1.6.0->apache-beam==2.43.0.dev0) (21.3)
Collecting google-resumable-media<3.0dev,>=0.6.0
Using cached google_resumable_media-2.4.0-py2.py3-none-any.whl (77 kB)
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
Using cached grpc_google_iam_v1-0.12.4-py2.py3-none-any.whl (26 kB)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.49.1-py3-none-any.whl (14 kB)
Collecting overrides<7.0.0,>=6.0.1
Using cached overrides-6.5.0-py3-none-any.whl (17 kB)
Collecting sqlparse>=0.3.0
Using cached sqlparse-0.4.3-py3-none-any.whl (42 kB)
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages> (from httplib2<0.21.0,>=0.8->apache-beam==2.43.0.dev0) (3.0.9)
Collecting attrs>=19.2.0
Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting exceptiongroup>=1.0.0rc8
Using cached exceptiongroup-1.0.0rc9-py3-none-any.whl (12 kB)
Collecting sortedcontainers<3.0.0,>=2.1.0
Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Collecting pbr>=0.11
Using cached pbr-5.10.0-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: pluggy<2.0,>=0.12 in <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages> (from pytest<8.0,>=7.1.2->apache-beam==2.43.0.dev0) (1.0.0)
Collecting iniconfig
Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Requirement already satisfied: py>=1.8.2 in <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages> (from pytest<8.0,>=7.1.2->apache-beam==2.43.0.dev0) (1.11.0)
Collecting tomli>=1.0.0
Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting execnet>=1.1
Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB)
Collecting pytest-forked
Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting charset-normalizer<3,>=2
Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
Collecting threadpoolctl>=2.0.0
Using cached threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting scipy>=1.3.2
Using cached scipy-1.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.8 MB)
Collecting greenlet!=0.4.17
Using cached greenlet-1.1.3.post0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (154 kB)
Collecting deprecation
Using cached deprecation-2.1.0-py2.py3-none-any.whl (11 kB)
Collecting wrapt
Using cached wrapt-1.14.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (77 kB)
Collecting docker>=4.0.0
Using cached docker-6.0.0-py3-none-any.whl (147 kB)
Collecting pymysql
Using cached PyMySQL-1.0.2-py3-none-any.whl (43 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting websocket-client>=0.32.0
Using cached websocket_client-1.4.1-py3-none-any.whl (55 kB)
Collecting googleapis-common-protos<2.0dev,>=1.56.2
Using cached googleapis_common_protos-1.56.4-py2.py3-none-any.whl (211 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (32 kB)
Collecting grpcio-status>=1.16.0
Using cached grpcio_status-1.48.2-py3-none-any.whl (14 kB)
Collecting isodate>=0.6.0
Using cached isodate-0.6.1-py2.py3-none-any.whl (41 kB)
Collecting requests-oauthlib>=0.5.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting pyasn1>=0.1.7
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.1-py3-none-any.whl (151 kB)
Building wheels for collected packages: apache-beam
Building wheel for apache-beam (setup.py): started
Building wheel for apache-beam (setup.py): finished with status 'done'
Created wheel for apache-beam: filename=apache_beam-2.43.0.dev0-py3-none-any.whl size=2873533 sha256=09f53ce7b61f36527e3ceefe2c946128bad05c017d1bd579238d012fb709c289
Stored in directory: /home/jenkins/.cache/pip/wheels/21/8c/b7/a45980b6c921f26c5ff58544c0ad1f681abebd49fb3b9260a4
Successfully built apache-beam
Installing collected packages: sortedcontainers, pytz, pyasn1, parameterized, iniconfig, docopt, crcmod, zstandard, wrapt, websocket-client, urllib3, typing-extensions, tomli, threadpoolctl, tenacity, sqlparse, scipy, rsa, regex, pyyaml, python-dateutil, pymysql, pymongo, pyhamcrest, pydot, pycparser, pyasn1-modules, pyarrow, psycopg2-binary, proto-plus, pbr, overrides, orjson, objsize, oauthlib, joblib, jmespath, isodate, idna, httplib2, greenlet, googleapis-common-protos, google-crc32c, fasteners, fastavro, execnet, exceptiongroup, dill, cloudpickle, charset-normalizer, certifi, cachetools, attrs, sqlalchemy, scikit-learn, requests, pytest, pandas, oauth2client, mock, hypothesis, grpcio-status, google-resumable-media, google-auth, freezegun, deprecation, cffi, botocore, s3transfer, requests-oauthlib, requests_mock, pytest-timeout, pytest-forked, hdfs, grpc-google-iam-v1, google-auth-httplib2, google-apitools, google-api-core, docker, cryptography, azure-core, testcontainers, pytest-xdist, msrest, google-cloud-core, boto3, apache-beam, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-language, google-cloud-dlp, google-cloud-datastore, google-cloud-bigtable, google-cloud-bigquery-storage, azure-storage-blob, google-cloud-pubsublite, google-cloud-bigquery
Successfully installed apache-beam-2.43.0.dev0 attrs-22.1.0 azure-core-1.26.0 azure-storage-blob-12.14.0 boto3-1.24.91 botocore-1.27.91 cachetools-4.2.4 certifi-2022.9.24 cffi-1.15.1 charset-normalizer-2.1.1 cloudpickle-2.2.0 crcmod-1.7 cryptography-38.0.1 deprecation-2.1.0 dill-0.3.1.1 docker-6.0.0 docopt-0.6.2 exceptiongroup-1.0.0rc9 execnet-1.9.0 fastavro-1.6.1 fasteners-0.18 freezegun-1.2.2 google-api-core-2.10.2 google-apitools-0.5.31 google-auth-2.12.0 google-auth-httplib2-0.1.0 google-cloud-bigquery-3.3.5 google-cloud-bigquery-storage-2.13.2 google-cloud-bigtable-1.7.2 google-cloud-core-2.3.2 google-cloud-datastore-1.15.5 google-cloud-dlp-3.9.2 google-cloud-language-1.3.2 google-cloud-pubsub-2.13.10 google-cloud-pubsublite-1.5.0 google-cloud-recommendations-ai-0.7.1 google-cloud-spanner-3.22.2 google-cloud-videointelligence-1.16.3 google-cloud-vision-1.0.2 google-crc32c-1.5.0 google-resumable-media-2.4.0 googleapis-common-protos-1.56.4 greenlet-1.1.3.post0 grpc-google-iam-v1-0.12.4 grpcio-status-1.48.2 hdfs-2.7.0 httplib2-0.20.4 hypothesis-6.56.2 idna-3.4 iniconfig-1.1.1 isodate-0.6.1 jmespath-1.0.1 joblib-1.2.0 mock-2.0.0 msrest-0.7.1 oauth2client-4.1.3 oauthlib-3.2.1 objsize-0.5.2 orjson-3.8.0 overrides-6.5.0 pandas-1.4.4 parameterized-0.8.1 pbr-5.10.0 proto-plus-1.22.1 psycopg2-binary-2.9.4 pyarrow-9.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pymysql-1.0.2 pytest-7.1.3 pytest-forked-1.4.0 pytest-timeout-2.1.0 pytest-xdist-2.5.0 python-dateutil-2.8.2 pytz-2022.4 pyyaml-6.0 regex-2022.9.13 requests-2.28.1 requests-oauthlib-1.3.1 requests_mock-1.10.0 rsa-4.9 s3transfer-0.6.0 scikit-learn-1.1.2 scipy-1.9.2 sortedcontainers-2.4.0 sqlalchemy-1.4.41 sqlparse-0.4.3 tenacity-5.1.5 testcontainers-3.7.0 threadpoolctl-3.1.0 tomli-2.0.1 typing-extensions-4.4.0 urllib3-1.26.12 websocket-client-1.4.1 wrapt-1.14.1 zstandard-0.18.0
> Task :sdks:python:test-suites:dataflow:py39:examples
>>> RUNNING integration tests with pipeline options: --runner=TestDataflowRunner --project=apache-beam-testing --region=us-central1 --staging_location=gs://temp-storage-for-end-to-end-tests/staging-it --temp_location=gs://temp-storage-for-end-to-end-tests/temp-it --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output --sdk_location=<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz> --requirements_file=postcommit_requirements.txt --num_workers=1 --sleep_secs=20 --experiments=use_runner_v2 --experiments=shuffle_mode=appliance --experiments=beam_fn_api --kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test --dataflow_kms_key=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test
>>> pytest options: --capture=no --timeout=4500 --color=yes --log-cli-level=INFO --numprocesses=8 --dist=loadfile
>>> collect markers: -m=examples_postcommit and not no_xdist and not sickbay_dataflow
[1m============================= test session starts ==============================[0m
platform linux -- Python 3.9.10, pytest-7.1.3, pluggy-1.0.0
rootdir: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python,> configfile: pytest.ini
plugins: xdist-2.5.0, timeout-2.1.0, hypothesis-6.56.2, forked-1.4.0, requests-mock-1.10.0
timeout: 4500.0s
timeout method: signal
timeout func_only: False
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I
[1m[gw0] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw1] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw2] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw3] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw4] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw5] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw6] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw7] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0mgw0 [18] / gw1 [18] / gw2 [18] / gw3 [18] / gw4 [18] / gw5 [18] / gw6 [18] / gw7 [18]
scheduling tests via LoadFileScheduling
apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py::JuliaSetTest::test_generate_fractal_image
apache_beam/examples/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/complete/tfidf_test.py::TfIdfTest::test_basics
apache_beam/examples/complete/distribopt_test.py::DistribOptimizationTest::test_basics
apache_beam/examples/wordcount_debugging_test.py::WordCountDebuggingTest::test_basics
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/wordcount_minimal_test.py::WordCountMinimalTest::test_basics
[gw7] [33mSKIPPED[0m apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py::JuliaSetTest::test_generate_fractal_image
apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py::JuliaSetTest::test_output_file_format
[gw3] [32mPASSED[0m apache_beam/examples/complete/distribopt_test.py::DistribOptimizationTest::test_basics
apache_beam/examples/cookbook/group_with_coder_test.py::GroupWithCoderTest::test_basics_with_type_check
[gw7] [32mPASSED[0m apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py::JuliaSetTest::test_output_file_format
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
[gw4] [32mPASSED[0m apache_beam/examples/complete/tfidf_test.py::TfIdfTest::test_basics
apache_beam/examples/cookbook/mergecontacts_test.py::MergeContactsTest::test_mergecontacts
[gw1] [32mPASSED[0m apache_beam/examples/wordcount_minimal_test.py::WordCountMinimalTest::test_basics
apache_beam/examples/cookbook/custom_ptransform_test.py::CustomCountTest::test_custom_ptransform_output_files_on_small_input
[gw2] [32mPASSED[0m apache_beam/examples/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
[gw0] [32mPASSED[0m apache_beam/examples/wordcount_debugging_test.py::WordCountDebuggingTest::test_basics
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
[gw7] [32mPASSED[0m apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
[gw3] [32mPASSED[0m apache_beam/examples/cookbook/group_with_coder_test.py::GroupWithCoderTest::test_basics_with_type_check
apache_beam/examples/cookbook/group_with_coder_test.py::GroupWithCoderTest::test_basics_without_type_check
[gw1] [32mPASSED[0m apache_beam/examples/cookbook/custom_ptransform_test.py::CustomCountTest::test_custom_ptransform_output_files_on_small_input
[gw4] [32mPASSED[0m apache_beam/examples/cookbook/mergecontacts_test.py::MergeContactsTest::test_mergecontacts
[gw3] [32mPASSED[0m apache_beam/examples/cookbook/group_with_coder_test.py::GroupWithCoderTest::test_basics_without_type_check
[gw0] [32mPASSED[0m apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
[gw2] node down: Not properly terminated
[gw2] [31mFAILED[0m apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher Terminated
The message received from the daemon indicates that the daemon has disappeared.
Build request sent: Build{id=9e35fbd7-a236-4c33-ba32-1da377b147f5, currentDir=<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src}>
Attempting to read last messages from the daemon log...
Daemon pid: 692026
log file: /home/jenkins/.gradle/daemon/7.5.1/daemon-692026.out.log
----- Last 20 lines from daemon log file - daemon-692026.out.log -----
[gw3] [32mPASSED[0m apache_beam/examples/cookbook/group_with_coder_test.py::GroupWithCoderTest::test_basics_without_type_check
[gw0] [32mPASSED[0m apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
[gw2] node down: Not properly terminated
[gw2] [31mFAILED[0m apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher Terminated
Daemon vm is shutting down... The daemon has exited normally or was terminated in response to a user interrupt.
Remove shutdown hook failed
java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
at org.gradle.process.internal.shutdown.ShutdownHooks.removeShutdownHook(ShutdownHooks.java:38)
at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:208)
at org.gradle.process.internal.DefaultExecHandle.aborted(DefaultExecHandle.java:366)
at org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunner.java:108)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:84)
at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
----- End of the daemon log -----
FAILURE: Build failed with an exception.
* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_Python_Examples_Dataflow #1024
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1024/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1023
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1023/display/redirect?page=changes>
Changes:
[Kenneth Knowles] Verify that secondary key coder is deterministic in SortValues
[riteshghorse] fix lints
[noreply] Update google cloud vision >= 2.0.0 (#23755)
------------------------------------------
[...truncated 178.19 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpkpuqnpum/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.44.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7faa141e4c10> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7faa141e53f0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7fa9bfb3df60>
http = <httplib2.Http object at 0x7fa9bfb3cca0>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7fa9bfb3d6c0>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F30870152-15cc-4123-a2c2-1e7d98dbc87f%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdvMtr6aGscwnoQh6-MT4UDk8N4A9WZmzmMJQi_U5XALKXaEeCbCUqqg9RqpHTWJlaKdwdMuj_YVJwBmVCzsbpg83lrhr_cJ', 'date': 'Fri, 21 Oct 2022 02:43:59 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/30870152-15cc-4123-a2c2-1e7d98dbc87f/input.txt
[31m[1mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/30870152-15cc-4123-a2c2-1e7d98dbc87f/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F30870152-15cc-4123-a2c2-1e7d98dbc87f%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdvMtr6aGscwnoQh6-MT4UDk8N4A9WZmzmMJQi_U5XALKXaEeCbCUqqg9RqpHTWJlaKdwdMuj_YVJwBmVCzsbpg83lrhr_cJ', 'date': 'Fri, 21 Oct 2022 02:43:59 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4877.28s (1:21:17)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 48m 35s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/gvofksc53qvie
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1022
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1022/display/redirect?page=changes>
Changes:
[noreply] Issue#23599 Updated dataframe notebook
[noreply] Added a missing line break.
[Chamikara Madhusanka Jayalath] Updating Python dependencies for the 2.43.0 release
[thiagotnunes] tests: fixes SpannerIO unavailable retry test
[noreply] Remove yeandy from reviewers (#23753)
[noreply] Revert bigdataoss version upgrade (#23727)
[Chamikara Madhusanka Jayalath] Moving to 2.44.0-SNAPSHOT on master branch.
[noreply] Update the timeout in ValidatesContainer suite. (#23732)
------------------------------------------
[...truncated 178.23 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmprcq20ibj/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.44.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f088ca50c10> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f088ca513f0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f08406a1b70>
http = <httplib2.Http object at 0x7f08406a2830>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f08406a2410>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fe3b00b6e-14eb-4968-bf85-bff01cae30c0%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdubcz4M8aE5nNR2MPhm6z9YcuFTVHJNLGniVczPB9M6PlUzues0rrTY3DNKdqr2YJ2rv28MYtz6S1XyhhZ1XfR7Rw', 'date': 'Thu, 20 Oct 2022 20:44:16 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/e3b00b6e-14eb-4968-bf85-bff01cae30c0/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/e3b00b6e-14eb-4968-bf85-bff01cae30c0/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fe3b00b6e-14eb-4968-bf85-bff01cae30c0%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdubcz4M8aE5nNR2MPhm6z9YcuFTVHJNLGniVczPB9M6PlUzues0rrTY3DNKdqr2YJ2rv28MYtz6S1XyhhZ1XfR7Rw', 'date': 'Thu, 20 Oct 2022 20:44:16 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4885.21s (1:21:25)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 48m 52s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/ufvwbjfkx7tbe
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1021
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1021/display/redirect>
Changes:
------------------------------------------
[...truncated 178.50 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp27v80qwl/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f032677cc10> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f032677d3f0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f02e5d22f50>
http = <httplib2.Http object at 0x7f02e5d226b0>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f02e5d22740>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F5aed1df9-36bb-476a-9fca-75758ad3fe70%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdssgOYOtXkuYkZx9e8dQPTry2dL3hu6gdMcwz--IFhyyq_EjSQOyr2xjqm6VtrcsQAJVtNa1-cqlxxjsJb6qwoUqQ', 'date': 'Thu, 20 Oct 2022 14:41:43 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/5aed1df9-36bb-476a-9fca-75758ad3fe70/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/5aed1df9-36bb-476a-9fca-75758ad3fe70/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F5aed1df9-36bb-476a-9fca-75758ad3fe70%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdssgOYOtXkuYkZx9e8dQPTry2dL3hu6gdMcwz--IFhyyq_EjSQOyr2xjqm6VtrcsQAJVtNa1-cqlxxjsJb6qwoUqQ', 'date': 'Thu, 20 Oct 2022 14:41:43 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4880.96s (1:21:20)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 46m 19s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/j55smgmquo7so
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1020
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1020/display/redirect?page=changes>
Changes:
[noreply] Reduce log flood in Python PostCommit flink task (#23635)
[noreply] Speed up check on website links (#23737)
------------------------------------------
[...truncated 178.68 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp2mfyl7_w/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f0bca714af0> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f0bca7152d0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f0b8ea02410>
http = <httplib2.Http object at 0x7f0b8ea02050>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f0b8ea00760>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fdcfe4e1b-54cf-4308-9d19-b5dbcb54ff9e%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycduEH56TeQ6AloypUYfRz2w7qQ9gjLHNMqCNxAoQfoiShmlV1Tr4599rR9rKhNsNNa9Z6yEP1evEzDDmN_SgGT6CfQ', 'date': 'Thu, 20 Oct 2022 08:47:03 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/dcfe4e1b-54cf-4308-9d19-b5dbcb54ff9e/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/dcfe4e1b-54cf-4308-9d19-b5dbcb54ff9e/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fdcfe4e1b-54cf-4308-9d19-b5dbcb54ff9e%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycduEH56TeQ6AloypUYfRz2w7qQ9gjLHNMqCNxAoQfoiShmlV1Tr4599rR9rKhNsNNa9Z6yEP1evEzDDmN_SgGT6CfQ', 'date': 'Thu, 20 Oct 2022 08:47:03 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4871.99s (1:21:11)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 51m 36s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/q5bjr5s3h3zmo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1019
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1019/display/redirect?page=changes>
Changes:
[noreply] Bump golang.org/x/text from 0.3.7 to 0.4.0 in /sdks (#23686)
[noreply] Unit Content markdown styles (#23592) (#23662)
[noreply] Add reopen issue command (#23733)
[noreply] Add example of real time Anomaly Detection using RunInference (#23497)
[noreply] Support TIMESTAMP type in BigQueryIO with BEAM_ROW output type, and in
[noreply] Add PytorchBatchConverter (#23296)
[noreply] Pin version to grpcio in build-requirements.txt (#23735)
[noreply] Bump up python container versions. (#23716)
------------------------------------------
[...truncated 178.46 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpajgp90w1/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7fdfe27a4af0> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7fdfe27a52d0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7fdfa6bb6410>
http = <httplib2.Http object at 0x7fdfa6bb7700>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7fdfa6bb7580>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fd979e278-4fc3-4310-bddb-48a0d1374370%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdumAPhS4thZRP-6n3ojKcIJgn6LalJUy6nvlGkbYCMR9F_Fn8q_bKqEJVx3l_MelDOR4VDY-hbQYopPFf6dTqj0xg', 'date': 'Thu, 20 Oct 2022 02:43:35 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/d979e278-4fc3-4310-bddb-48a0d1374370/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/d979e278-4fc3-4310-bddb-48a0d1374370/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fd979e278-4fc3-4310-bddb-48a0d1374370%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdumAPhS4thZRP-6n3ojKcIJgn6LalJUy6nvlGkbYCMR9F_Fn8q_bKqEJVx3l_MelDOR4VDY-hbQYopPFf6dTqj0xg', 'date': 'Thu, 20 Oct 2022 02:43:35 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4871.23s (1:21:11)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 48m 10s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/maxuan4zcn2ku
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1018
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1018/display/redirect?page=changes>
Changes:
[Moritz Mack] Keep Spark version in a single place only (BeamModulePlugin)
[noreply] Update release instructions in Python 3.10 (#23702)
[noreply] Move Tensorflow Documentation (#23729)
------------------------------------------
[...truncated 192.91 KB...]
File "apache_beam/runners/worker/operations.py", line 943, in apache_beam.runners.worker.operations.DoOperation.finish
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-b887561e-5303-4bd5-bb01-59fc1903e298/output.csv/[ComputedExpression[apply_DataFrame_140492028164128]]:140492026746272/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:17:40.164Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:17:40.240Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:17:40.264Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:19:55.697Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:19:55.745Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:19:55.773Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-19_12_11_25-10958520690669076923 is in state JOB_STATE_FAILED
[31m[1mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-19_12_11_25-10958520690669076923?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 2 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m1 failed[0m, [32m16 passed[0m, [33m4 skipped[0m, [33m98 warnings[0m[31m in 1358.38s (0:22:38)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 25m 16s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/jzd3nujt5xpfc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1017
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1017/display/redirect?page=changes>
Changes:
[noreply] [Playground] Examples CD (#23664)
------------------------------------------
[...truncated 192.71 KB...]
File "apache_beam/runners/worker/operations.py", line 943, in apache_beam.runners.worker.operations.DoOperation.finish
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-9265b9d1-1e82-4bc7-94c9-56e1fab30e88/output.csv/[ComputedExpression[apply_DataFrame_139855937218128]]:139856149281808/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:21:23.735Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:21:23.822Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:21:23.853Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:23:43.919Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:23:43.999Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:23:44.028Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-19_06_12_11-8793419384548762595 is in state JOB_STATE_FAILED
[31m[1mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-19_06_12_11-8793419384548762595?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 2 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m1 failed[0m, [32m16 passed[0m, [33m4 skipped[0m, [33m98 warnings[0m[31m in 1605.44s (0:26:45)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 29m 4s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/rqpfzdzjedz46
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1016
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1016/display/redirect?page=changes>
Changes:
[noreply] Merge pull request #23510: Vortex multiplexing streams
[noreply] Io jms fix ack message checkpoint (#22932)
------------------------------------------
[...truncated 196.81 KB...]
File "apache_beam/runners/worker/operations.py", line 943, in apache_beam.runners.worker.operations.DoOperation.finish
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-8f52b439-cf85-49c1-bc14-e762a2869993/output.csv/[ComputedExpression[apply_DataFrame_140056269315792]]:140056267888272/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:22:10.047Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:22:10.121Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:22:10.158Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:24:25.909Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:24:25.965Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:24:25.995Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-19_00_12_48-7657686204605454511 is in state JOB_STATE_FAILED
[1m[31mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-19_00_12_48-7657686204605454511?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 2 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m1 failed[0m, [32m16 passed[0m, [33m4 skipped[0m, [33m98 warnings[0m[31m in 1632.00s (0:27:12)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 29m 41s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/57tqkffgf4kns
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1015
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1015/display/redirect?page=changes>
Changes:
[Robert Bradshaw] Remove numpy C API dep from public declarations.
[Kiley Sok] beam-perf
[Kiley Sok] fix
[noreply] Add java 11 home to jenkins test (#23708)
[noreply] enable automatic expansion service (#23699)
[noreply] add expansion service option (#23712)
[noreply] Downgrade container cryptography version to avoid yanked version
[noreply] Update portable runner test timeout (#23696)
------------------------------------------
[...truncated 192.87 KB...]
File "apache_beam/runners/worker/operations.py", line 943, in apache_beam.runners.worker.operations.DoOperation.finish
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-50724c18-b2b2-4dcc-906e-9fcd7667613e/output.csv/[ComputedExpression[apply_DataFrame_140313415021472]]:140313416860288/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:18:58.676Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:18:58.788Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:18:58.813Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:21:11.292Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:21:11.340Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:21:11.373Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-18_18_11_29-6577956473634859488 is in state JOB_STATE_FAILED
[1m[31mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-18_18_11_29-6577956473634859488?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 2 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m1 failed[0m, [32m16 passed[0m, [33m4 skipped[0m, [33m98 warnings[0m[31m in 1455.76s (0:24:15)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 26m 27s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/dk2zzo7s24zjo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1014
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1014/display/redirect?page=changes>
Changes:
[Andrew Pilloud] Migrate nexmark to common config for cron jobs
[noreply] [Go SDK] Dataframe API wrapper (#23450)
[noreply] [Go SDK]: Adds Automated Python Expansion Service (#23582)
[noreply] Include CombineFn's in __all__ (#23685)
[noreply] Bump google.golang.org/grpc from 1.50.0 to 1.50.1 in /sdks (#23654)
[noreply] [Playground][Frontend] Tags filter for Examples Catalog (#22074)
[noreply] [Go SDK] Extract output coders in expandCrossLanguage (#23641)
[noreply] Python 3.10 support (#23587)
[noreply] Fixes #22192: Avoids nullpointer error. Preserves previous behavior.
[noreply] Deflaking tests for BQ row insertions. These tests were flaky due to
------------------------------------------
[...truncated 261.35 KB...]
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-f9a7f483-a708-404c-bfa7-41d2db50d9f3/output.csv/[ComputedExpression[apply_DataFrame_140679440005824]]:140679440571056/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:00:07.399Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:00:07.730Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:00:07.755Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:02:25.918Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:02:26.008Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:02:26.043Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-18_12_53_37-8879752588403415834 is in state JOB_STATE_FAILED
[31m[1mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-18_12_53_37-8879752588403415834?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 3 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m2 failed[0m, [32m15 passed[0m, [33m4 skipped[0m, [33m99 warnings[0m[31m in 1366.85s (0:22:46)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 28m 34s
15 actionable tasks: 15 executed
Publishing build scan...
https://gradle.com/s/pq7yzjyapbdds
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1013
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1013/display/redirect?page=changes>
Changes:
[Alexey Romanenko] [TPC-DS] Use "nonpartitioned" input for Jenkins jobs
[noreply] [GitHub Actions] - Run RC Validations Workflow (#23531)
[noreply] Add workflow to update milestone on issue close (#23629)
[noreply] add website page about data processing for ML (#23552)
------------------------------------------
[...truncated 174.27 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7f866d732a00>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7f86455d5040>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp81y60g5t/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f863635a550> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f863635ad30> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f866ddd5eb0>
http = <httplib2.Http object at 0x7f866ddd3580>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f86372035e0>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F9de38bd9-3148-4484-b337-ff225198c3fb%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdvKvsSygzinK1PKTtfC7uLMVf0aN43WRwa5F0nk49SJzchB8Dsnne8hILbkP8uxlw0kiruNJmiik-_o2ZffO5u7Mg', 'date': 'Tue, 18 Oct 2022 14:42:47 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/9de38bd9-3148-4484-b337-ff225198c3fb/input.txt
[31m[1mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/9de38bd9-3148-4484-b337-ff225198c3fb/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F9de38bd9-3148-4484-b337-ff225198c3fb%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdvKvsSygzinK1PKTtfC7uLMVf0aN43WRwa5F0nk49SJzchB8Dsnne8hILbkP8uxlw0kiruNJmiik-_o2ZffO5u7Mg', 'date': 'Tue, 18 Oct 2022 14:42:47 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4871.66s (1:21:11)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 47m 25s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/5vcp7gq5gizeg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1012
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1012/display/redirect?page=changes>
Changes:
[noreply] Fix python log_level_overrides cannot be used on flink and other
[noreply] Better error for disabling runner v2 with cross language pipelines.
[noreply] Update scopes to match the existing set and update test to clear
------------------------------------------
[...truncated 174.22 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7f3df280e250>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7f3dc66e8040>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp1s08o42l/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f3da345e550> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f3da345ed30> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f3df28d7ee0>
http = <httplib2.Http object at 0x7f3df2eeed90>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f3df2eeed00>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F12c5e1a1-6fd7-40a3-b429-11f6ef6912b7%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdt63FdVsNADID97JknjSQ3-hTj5iBp0OMMBrWINXBP5b9I8mJaaIGFEMuX2a4OLbCDRvPpF1Xy4nDozZfWle5Fpd3GZYAgk', 'date': 'Tue, 18 Oct 2022 08:45:14 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/12c5e1a1-6fd7-40a3-b429-11f6ef6912b7/input.txt
[31m[1mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/12c5e1a1-6fd7-40a3-b429-11f6ef6912b7/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F12c5e1a1-6fd7-40a3-b429-11f6ef6912b7%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdt63FdVsNADID97JknjSQ3-hTj5iBp0OMMBrWINXBP5b9I8mJaaIGFEMuX2a4OLbCDRvPpF1Xy4nDozZfWle5Fpd3GZYAgk', 'date': 'Tue, 18 Oct 2022 08:45:14 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4877.65s (1:21:17)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 49m 52s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/73rgwnljeuuom
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1011
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1011/display/redirect?page=changes>
Changes:
[noreply] Fixes #23627: Speed up website checks (#23673)
[noreply] Suppress a FloatingPointLiteralPrecision error (#23667)
[noreply] Improved test coverage and fix the implementation of Inject and CoGBK
------------------------------------------
[...truncated 174.22 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7fc890de1220>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7fc868c40ee0>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp4eqf_bl3/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:480 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7fc849993430> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7fc849993c10> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7fc895c15910>
http = <httplib2.Http object at 0x7fc84a84a100>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7fc89145c370>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fca5c6192-9dbf-4296-9935-6f8fc29557b4%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdtxPi0VR07bC5ezg4ZMo1Atyskd2dLOHvbeJ8WnpwyG4uWwjrpoIk0iTCYnvLvVeRSyfpr-_IR1pVj4fG9Gpqp7uoRkg5Mz', 'date': 'Tue, 18 Oct 2022 02:37:28 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/ca5c6192-9dbf-4296-9935-6f8fc29557b4/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/ca5c6192-9dbf-4296-9935-6f8fc29557b4/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fca5c6192-9dbf-4296-9935-6f8fc29557b4%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdtxPi0VR07bC5ezg4ZMo1Atyskd2dLOHvbeJ8WnpwyG4uWwjrpoIk0iTCYnvLvVeRSyfpr-_IR1pVj4fG9Gpqp7uoRkg5Mz', 'date': 'Tue, 18 Oct 2022 02:37:28 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4872.23s (1:21:12)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 42m 5s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/losvbzftmzkoa
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1010
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1010/display/redirect?page=changes>
Changes:
[Moritz Mack] [Jenkins,Spark] Stop running Nexmark suite for deprecated Spark 2 runner
[noreply] [Tour Of Beam] README update (#23318)
[noreply] Bump google.golang.org/api from 0.98.0 to 0.99.0 in /sdks (#23655)
[noreply] Fix beam_PerformanceTests_PubsubIOIT_Python_Streaming (#23607)
[noreply] 2.42.0 Release Blog Post (#23406)
[noreply] Docs for state in go (#22965)
[noreply] Fix typo in 2.42.0 blog.
[noreply] Adjust 2.42.0 publishing time.
[noreply] Adds a Java RunInference example (#23619)
------------------------------------------
[...truncated 174.80 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7efdf19a6ac0>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7efdc5889ee0>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpb2ws9qmf/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:480 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7efdaae11430> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7efdaae11c10> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7efdf65f7a60>
http = <httplib2.Http object at 0x7efdf20898e0>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7efdf6898af0>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fe27653ee-40ac-47f6-ae7a-70e16e314e06%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdu2gnwg3_A1laKBQJk3sGl1FUd_3ZSkeoQpu5HfddbjM0gMWBtHOpIemjny2mIxgM26eF3-5BjQvrF8RDdDdCL6Lw', 'date': 'Mon, 17 Oct 2022 21:09:53 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/e27653ee-40ac-47f6-ae7a-70e16e314e06/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/e27653ee-40ac-47f6-ae7a-70e16e314e06/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fe27653ee-40ac-47f6-ae7a-70e16e314e06%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdu2gnwg3_A1laKBQJk3sGl1FUd_3ZSkeoQpu5HfddbjM0gMWBtHOpIemjny2mIxgM26eF3-5BjQvrF8RDdDdCL6Lw', 'date': 'Mon, 17 Oct 2022 21:09:53 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4881.90s (1:21:21)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 53m 15s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/ekccpvxv6yg7w
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1009
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1009/display/redirect?page=changes>
Changes:
[noreply] [Playground] Examples CI (#23476)
------------------------------------------
[...truncated 175.03 KB...]
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmph31ul6bf/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:480 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7fd4bb8a1310> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7fd4bb8a1af0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7fd4f2b4a400>
http = <httplib2.Http object at 0x7fd4f28d3220>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7fd4f28d3040>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F9422931d-21f4-480c-bb25-4de00d084847%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdsb_Ho6cXc7j8o6t9f88HUfR0NZX-V7wH1eEETmcPQU3LUr6NWkzwVNaMEFzvi13ZijWisC_WIeHzIm8MkKHPAcCw', 'date': 'Mon, 17 Oct 2022 14:46:03 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/9422931d-21f4-480c-bb25-4de00d084847/input.txt
[31m[1mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/9422931d-21f4-480c-bb25-4de00d084847/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F9422931d-21f4-480c-bb25-4de00d084847%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdsb_Ho6cXc7j8o6t9f88HUfR0NZX-V7wH1eEETmcPQU3LUr6NWkzwVNaMEFzvi13ZijWisC_WIeHzIm8MkKHPAcCw', 'date': 'Mon, 17 Oct 2022 14:46:03 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4875.10s (1:21:15)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 50m 37s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
Publishing build scan failed due to network error 'java.net.SocketException: Unexpected end of file from server' (2 retries remaining)...
Publishing build scan failed due to network error 'java.net.SocketException: Unexpected end of file from server' (1 retry remaining)...
A network error occurred.
If you require assistance with this problem, please report it via https://gradle.com/help/plugin and include the following information via copy/paste.
----------
Gradle version: 7.5.1
Plugin version: 3.4.1
Request URL: https://status.gradle.com
Request ID: 72195274-651a-41e6-9f39-e52e93c939d4
Exception: java.net.SocketException: Unexpected end of file from server
----------
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1008
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1008/display/redirect>
Changes:
------------------------------------------
[...truncated 176.39 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7f06405162b0>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7f06182f7ee0>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpvc76bagu/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:480 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f060105c310> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f060105caf0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f0644d916a0>
http = <httplib2.Http object at 0x7f0640a58760>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f0644d91b80>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Ff9803341-8d85-4603-8cf9-3518c51fd269%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycds3BypCfr7LVvifdHDQ5v2E7FQsl6Z6OA79wxV39UQcUXV3iBL3GzuujUeTNq_ifvOI5pF-8GFpZviGicXMS_pMHQ', 'date': 'Mon, 17 Oct 2022 08:43:54 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/f9803341-8d85-4603-8cf9-3518c51fd269/input.txt
[31m[1mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/f9803341-8d85-4603-8cf9-3518c51fd269/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Ff9803341-8d85-4603-8cf9-3518c51fd269%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycds3BypCfr7LVvifdHDQ5v2E7FQsl6Z6OA79wxV39UQcUXV3iBL3GzuujUeTNq_ifvOI5pF-8GFpZviGicXMS_pMHQ', 'date': 'Mon, 17 Oct 2022 08:43:54 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4874.70s (1:21:14)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 48m 29s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/pde3hfzlnc7yy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1007
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1007/display/redirect?page=changes>
Changes:
[noreply] Blog post for Hop web in Google Cloud (#23652)
------------------------------------------
[...truncated 174.25 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7f653ce156d0>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7f6514bf5ee0>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpfq5yfveu/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:480 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f6506159310> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f6506159af0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f653d19d100>
http = <httplib2.Http object at 0x7f6506fb93d0>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f654158f370>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fa2621961-0016-4614-b9a4-a866ea4f9be9%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdu2fO7JS8WhBxWTqR8Q4zc3PHyxMsoJ6qztMsQireZt0Lvt2WhRgBtBeJtzcemDPsjeDkuoGHjycvcYuCNVMEZ3-94hZOFL', 'date': 'Mon, 17 Oct 2022 02:44:10 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/a2621961-0016-4614-b9a4-a866ea4f9be9/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/a2621961-0016-4614-b9a4-a866ea4f9be9/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fa2621961-0016-4614-b9a4-a866ea4f9be9%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdu2fO7JS8WhBxWTqR8Q4zc3PHyxMsoJ6qztMsQireZt0Lvt2WhRgBtBeJtzcemDPsjeDkuoGHjycvcYuCNVMEZ3-94hZOFL', 'date': 'Mon, 17 Oct 2022 02:44:10 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4870.53s (1:21:10)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 48m 44s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/rhgr3srn77tme
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1006
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1006/display/redirect>
Changes:
------------------------------------------
[...truncated 174.06 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7fec0044aa90>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7febd82ebee0>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpmankhlvv/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:480 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7febc905d310> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7febc905daf0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7fec04d07ee0>
http = <httplib2.Http object at 0x7fec004dc9d0>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7fec004dc730>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F12d859c6-4401-4ef2-bf04-13c726bc9e11%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycds7s1EgXQz2Dc211tbVJelTTi3ZBmKMvHklejrE_xLn_lAQ-7hEwRr6DxZCoy880yLESFY1-IJrEVD5h0i_vyTqUQ', 'date': 'Sun, 16 Oct 2022 20:41:56 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/12d859c6-4401-4ef2-bf04-13c726bc9e11/input.txt
[31m[1mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/12d859c6-4401-4ef2-bf04-13c726bc9e11/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F12d859c6-4401-4ef2-bf04-13c726bc9e11%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycds7s1EgXQz2Dc211tbVJelTTi3ZBmKMvHklejrE_xLn_lAQ-7hEwRr6DxZCoy880yLESFY1-IJrEVD5h0i_vyTqUQ', 'date': 'Sun, 16 Oct 2022 20:41:56 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4870.46s (1:21:10)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 46m 34s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/2i2nxpkwxeswa
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org