You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2022/10/17 21:09:57 UTC
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1010
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1010/display/redirect?page=changes>
Changes:
[Moritz Mack] [Jenkins,Spark] Stop running Nexmark suite for deprecated Spark 2 runner
[noreply] [Tour Of Beam] README update (#23318)
[noreply] Bump google.golang.org/api from 0.98.0 to 0.99.0 in /sdks (#23655)
[noreply] Fix beam_PerformanceTests_PubsubIOIT_Python_Streaming (#23607)
[noreply] 2.42.0 Release Blog Post (#23406)
[noreply] Docs for state in go (#22965)
[noreply] Fix typo in 2.42.0 blog.
[noreply] Adjust 2.42.0 publishing time.
[noreply] Adds a Java RunInference example (#23619)
------------------------------------------
[...truncated 174.80 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7efdf19a6ac0>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7efdc5889ee0>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpb2ws9qmf/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:480 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7efdaae11430> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7efdaae11c10> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7efdf65f7a60>
http = <httplib2.Http object at 0x7efdf20898e0>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7efdf6898af0>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fe27653ee-40ac-47f6-ae7a-70e16e314e06%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdu2gnwg3_A1laKBQJk3sGl1FUd_3ZSkeoQpu5HfddbjM0gMWBtHOpIemjny2mIxgM26eF3-5BjQvrF8RDdDdCL6Lw', 'date': 'Mon, 17 Oct 2022 21:09:53 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/e27653ee-40ac-47f6-ae7a-70e16e314e06/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/e27653ee-40ac-47f6-ae7a-70e16e314e06/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fe27653ee-40ac-47f6-ae7a-70e16e314e06%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdu2gnwg3_A1laKBQJk3sGl1FUd_3ZSkeoQpu5HfddbjM0gMWBtHOpIemjny2mIxgM26eF3-5BjQvrF8RDdDdCL6Lw', 'date': 'Mon, 17 Oct 2022 21:09:53 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4881.90s (1:21:21)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 53m 15s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/ekccpvxv6yg7w
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_Python_Examples_Dataflow #1024
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1024/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1023
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1023/display/redirect?page=changes>
Changes:
[Kenneth Knowles] Verify that secondary key coder is deterministic in SortValues
[riteshghorse] fix lints
[noreply] Update google cloud vision >= 2.0.0 (#23755)
------------------------------------------
[...truncated 178.19 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpkpuqnpum/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.44.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7faa141e4c10> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7faa141e53f0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7fa9bfb3df60>
http = <httplib2.Http object at 0x7fa9bfb3cca0>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7fa9bfb3d6c0>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F30870152-15cc-4123-a2c2-1e7d98dbc87f%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdvMtr6aGscwnoQh6-MT4UDk8N4A9WZmzmMJQi_U5XALKXaEeCbCUqqg9RqpHTWJlaKdwdMuj_YVJwBmVCzsbpg83lrhr_cJ', 'date': 'Fri, 21 Oct 2022 02:43:59 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/30870152-15cc-4123-a2c2-1e7d98dbc87f/input.txt
[31m[1mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/30870152-15cc-4123-a2c2-1e7d98dbc87f/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F30870152-15cc-4123-a2c2-1e7d98dbc87f%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdvMtr6aGscwnoQh6-MT4UDk8N4A9WZmzmMJQi_U5XALKXaEeCbCUqqg9RqpHTWJlaKdwdMuj_YVJwBmVCzsbpg83lrhr_cJ', 'date': 'Fri, 21 Oct 2022 02:43:59 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4877.28s (1:21:17)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 48m 35s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/gvofksc53qvie
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1022
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1022/display/redirect?page=changes>
Changes:
[noreply] Issue#23599 Updated dataframe notebook
[noreply] Added a missing line break.
[Chamikara Madhusanka Jayalath] Updating Python dependencies for the 2.43.0 release
[thiagotnunes] tests: fixes SpannerIO unavailable retry test
[noreply] Remove yeandy from reviewers (#23753)
[noreply] Revert bigdataoss version upgrade (#23727)
[Chamikara Madhusanka Jayalath] Moving to 2.44.0-SNAPSHOT on master branch.
[noreply] Update the timeout in ValidatesContainer suite. (#23732)
------------------------------------------
[...truncated 178.23 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmprcq20ibj/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.44.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f088ca50c10> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f088ca513f0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f08406a1b70>
http = <httplib2.Http object at 0x7f08406a2830>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f08406a2410>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fe3b00b6e-14eb-4968-bf85-bff01cae30c0%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdubcz4M8aE5nNR2MPhm6z9YcuFTVHJNLGniVczPB9M6PlUzues0rrTY3DNKdqr2YJ2rv28MYtz6S1XyhhZ1XfR7Rw', 'date': 'Thu, 20 Oct 2022 20:44:16 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/e3b00b6e-14eb-4968-bf85-bff01cae30c0/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/e3b00b6e-14eb-4968-bf85-bff01cae30c0/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fe3b00b6e-14eb-4968-bf85-bff01cae30c0%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdubcz4M8aE5nNR2MPhm6z9YcuFTVHJNLGniVczPB9M6PlUzues0rrTY3DNKdqr2YJ2rv28MYtz6S1XyhhZ1XfR7Rw', 'date': 'Thu, 20 Oct 2022 20:44:16 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4885.21s (1:21:25)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 48m 52s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/ufvwbjfkx7tbe
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1021
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1021/display/redirect>
Changes:
------------------------------------------
[...truncated 178.50 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp27v80qwl/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f032677cc10> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f032677d3f0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f02e5d22f50>
http = <httplib2.Http object at 0x7f02e5d226b0>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f02e5d22740>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F5aed1df9-36bb-476a-9fca-75758ad3fe70%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdssgOYOtXkuYkZx9e8dQPTry2dL3hu6gdMcwz--IFhyyq_EjSQOyr2xjqm6VtrcsQAJVtNa1-cqlxxjsJb6qwoUqQ', 'date': 'Thu, 20 Oct 2022 14:41:43 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/5aed1df9-36bb-476a-9fca-75758ad3fe70/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/5aed1df9-36bb-476a-9fca-75758ad3fe70/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F5aed1df9-36bb-476a-9fca-75758ad3fe70%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdssgOYOtXkuYkZx9e8dQPTry2dL3hu6gdMcwz--IFhyyq_EjSQOyr2xjqm6VtrcsQAJVtNa1-cqlxxjsJb6qwoUqQ', 'date': 'Thu, 20 Oct 2022 14:41:43 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4880.96s (1:21:20)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 46m 19s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/j55smgmquo7so
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1020
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1020/display/redirect?page=changes>
Changes:
[noreply] Reduce log flood in Python PostCommit flink task (#23635)
[noreply] Speed up check on website links (#23737)
------------------------------------------
[...truncated 178.68 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp2mfyl7_w/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f0bca714af0> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f0bca7152d0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f0b8ea02410>
http = <httplib2.Http object at 0x7f0b8ea02050>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f0b8ea00760>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fdcfe4e1b-54cf-4308-9d19-b5dbcb54ff9e%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycduEH56TeQ6AloypUYfRz2w7qQ9gjLHNMqCNxAoQfoiShmlV1Tr4599rR9rKhNsNNa9Z6yEP1evEzDDmN_SgGT6CfQ', 'date': 'Thu, 20 Oct 2022 08:47:03 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/dcfe4e1b-54cf-4308-9d19-b5dbcb54ff9e/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/dcfe4e1b-54cf-4308-9d19-b5dbcb54ff9e/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fdcfe4e1b-54cf-4308-9d19-b5dbcb54ff9e%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycduEH56TeQ6AloypUYfRz2w7qQ9gjLHNMqCNxAoQfoiShmlV1Tr4599rR9rKhNsNNa9Z6yEP1evEzDDmN_SgGT6CfQ', 'date': 'Thu, 20 Oct 2022 08:47:03 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4871.99s (1:21:11)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 51m 36s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/q5bjr5s3h3zmo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1019
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1019/display/redirect?page=changes>
Changes:
[noreply] Bump golang.org/x/text from 0.3.7 to 0.4.0 in /sdks (#23686)
[noreply] Unit Content markdown styles (#23592) (#23662)
[noreply] Add reopen issue command (#23733)
[noreply] Add example of real time Anomaly Detection using RunInference (#23497)
[noreply] Support TIMESTAMP type in BigQueryIO with BEAM_ROW output type, and in
[noreply] Add PytorchBatchConverter (#23296)
[noreply] Pin version to grpcio in build-requirements.txt (#23735)
[noreply] Bump up python container versions. (#23716)
------------------------------------------
[...truncated 178.46 KB...]
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/bin/python3.10',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpajgp90w1/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp310', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.10_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python310-fnapi:beam-master-20221018" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7fdfe27a4af0> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7fdfe27a52d0> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:50: in create_content_input_file
with gcs.open(path, 'w') as f:
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7fdfa6bb6410>
http = <httplib2.Http object at 0x7fdfa6bb7700>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7fdfa6bb7580>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fd979e278-4fc3-4310-bddb-48a0d1374370%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdumAPhS4thZRP-6n3ojKcIJgn6LalJUy6nvlGkbYCMR9F_Fn8q_bKqEJVx3l_MelDOR4VDY-hbQYopPFf6dTqj0xg', 'date': 'Thu, 20 Oct 2022 02:43:35 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/d979e278-4fc3-4310-bddb-48a0d1374370/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/d979e278-4fc3-4310-bddb-48a0d1374370/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fd979e278-4fc3-4310-bddb-48a0d1374370%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdumAPhS4thZRP-6n3ojKcIJgn6LalJUy6nvlGkbYCMR9F_Fn8q_bKqEJVx3l_MelDOR4VDY-hbQYopPFf6dTqj0xg', 'date': 'Thu, 20 Oct 2022 02:43:35 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/autocomplete_it_test.py: 1 warning
apache_beam/examples/complete/game/hourly_team_score_it_test.py: 2 warnings
apache_beam/examples/complete/game/user_score_it_test.py: 6 warnings
apache_beam/examples/cookbook/bigquery_side_input_it_test.py: 4 warnings
apache_beam/examples/cookbook/coders_it_test.py: 1 warning
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m4 skipped[0m, [33m6650 deselected[0m, [33m31 warnings[0m, [31m[1m2 errors[0m[31m in 4871.23s (1:21:11)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 48m 10s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/maxuan4zcn2ku
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1018
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1018/display/redirect?page=changes>
Changes:
[Moritz Mack] Keep Spark version in a single place only (BeamModulePlugin)
[noreply] Update release instructions in Python 3.10 (#23702)
[noreply] Move Tensorflow Documentation (#23729)
------------------------------------------
[...truncated 192.91 KB...]
File "apache_beam/runners/worker/operations.py", line 943, in apache_beam.runners.worker.operations.DoOperation.finish
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-b887561e-5303-4bd5-bb01-59fc1903e298/output.csv/[ComputedExpression[apply_DataFrame_140492028164128]]:140492026746272/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:17:40.164Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:17:40.240Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:17:40.264Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:19:55.697Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:19:55.745Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T19:19:55.773Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-19_12_11_25-10958520690669076923 is in state JOB_STATE_FAILED
[31m[1mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-19_12_11_25-10958520690669076923?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 2 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m1 failed[0m, [32m16 passed[0m, [33m4 skipped[0m, [33m98 warnings[0m[31m in 1358.38s (0:22:38)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 25m 16s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/jzd3nujt5xpfc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1017
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1017/display/redirect?page=changes>
Changes:
[noreply] [Playground] Examples CD (#23664)
------------------------------------------
[...truncated 192.71 KB...]
File "apache_beam/runners/worker/operations.py", line 943, in apache_beam.runners.worker.operations.DoOperation.finish
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-9265b9d1-1e82-4bc7-94c9-56e1fab30e88/output.csv/[ComputedExpression[apply_DataFrame_139855937218128]]:139856149281808/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:21:23.735Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:21:23.822Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:21:23.853Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:23:43.919Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:23:43.999Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T13:23:44.028Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-19_06_12_11-8793419384548762595 is in state JOB_STATE_FAILED
[31m[1mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-19_06_12_11-8793419384548762595?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 2 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m1 failed[0m, [32m16 passed[0m, [33m4 skipped[0m, [33m98 warnings[0m[31m in 1605.44s (0:26:45)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 29m 4s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/rqpfzdzjedz46
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1016
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1016/display/redirect?page=changes>
Changes:
[noreply] Merge pull request #23510: Vortex multiplexing streams
[noreply] Io jms fix ack message checkpoint (#22932)
------------------------------------------
[...truncated 196.81 KB...]
File "apache_beam/runners/worker/operations.py", line 943, in apache_beam.runners.worker.operations.DoOperation.finish
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-8f52b439-cf85-49c1-bc14-e762a2869993/output.csv/[ComputedExpression[apply_DataFrame_140056269315792]]:140056267888272/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:22:10.047Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:22:10.121Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:22:10.158Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:24:25.909Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:24:25.965Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T07:24:25.995Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-19_00_12_48-7657686204605454511 is in state JOB_STATE_FAILED
[1m[31mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-19_00_12_48-7657686204605454511?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 2 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m1 failed[0m, [32m16 passed[0m, [33m4 skipped[0m, [33m98 warnings[0m[31m in 1632.00s (0:27:12)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 29m 41s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/57tqkffgf4kns
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1015
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1015/display/redirect?page=changes>
Changes:
[Robert Bradshaw] Remove numpy C API dep from public declarations.
[Kiley Sok] beam-perf
[Kiley Sok] fix
[noreply] Add java 11 home to jenkins test (#23708)
[noreply] enable automatic expansion service (#23699)
[noreply] add expansion service option (#23712)
[noreply] Downgrade container cryptography version to avoid yanked version
[noreply] Update portable runner test timeout (#23696)
------------------------------------------
[...truncated 192.87 KB...]
File "apache_beam/runners/worker/operations.py", line 943, in apache_beam.runners.worker.operations.DoOperation.finish
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-50724c18-b2b2-4dcc-906e-9fcd7667613e/output.csv/[ComputedExpression[apply_DataFrame_140313415021472]]:140313416860288/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:18:58.676Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:18:58.788Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:18:58.813Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:21:11.292Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:21:11.340Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-19T01:21:11.373Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-18_18_11_29-6577956473634859488 is in state JOB_STATE_FAILED
[1m[31mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-18_18_11_29-6577956473634859488?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 2 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m1 failed[0m, [32m16 passed[0m, [33m4 skipped[0m, [33m98 warnings[0m[31m in 1455.76s (0:24:15)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 26m 27s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/dk2zzo7s24zjo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1014
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1014/display/redirect?page=changes>
Changes:
[Andrew Pilloud] Migrate nexmark to common config for cron jobs
[noreply] [Go SDK] Dataframe API wrapper (#23450)
[noreply] [Go SDK]: Adds Automated Python Expansion Service (#23582)
[noreply] Include CombineFn's in __all__ (#23685)
[noreply] Bump google.golang.org/grpc from 1.50.0 to 1.50.1 in /sdks (#23654)
[noreply] [Playground][Frontend] Tags filter for Examples Catalog (#22074)
[noreply] [Go SDK] Extract output coders in expandCrossLanguage (#23641)
[noreply] Python 3.10 support (#23587)
[noreply] Fixes #22192: Avoids nullpointer error. Preserves previous behavior.
[noreply] Deflaking tests for BQ row insertions. These tests were flaky due to
------------------------------------------
[...truncated 261.35 KB...]
File "apache_beam/runners/common.py", line 1479, in apache_beam.runners.common.DoFnRunner.finish
File "apache_beam/runners/common.py", line 1460, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1458, in apache_beam.runners.common.DoFnRunner._invoke_bundle_method
File "apache_beam/runners/common.py", line 561, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 566, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
File "apache_beam/runners/common.py", line 1745, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/transforms.py",> line -1, in evaluate
File "/usr/local/lib/python3.10/site-packages/apache_beam/dataframe/expressions.py", line 375, in evaluate_at
return self._func(*(session.evaluate(arg) for arg in self._args))
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in do_partition_apply
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/frames.py",> line -1, in <lambda>
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1425, in apply
return self._python_apply_general(f, self._selected_obj)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
values, mutated = self.grouper.apply(f, data, self.axis)
File "/usr/local/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 761, in apply
res = f(group)
File "/usr/local/lib/python3.10/site-packages/apache_beam/examples/dataframe/flight_delays.py", line 47, in get_mean_delay_at_top_airports
return airline_df[at_top_airports].mean()
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 11127, in mean
return NDFrame.mean(self, axis, skipna, level, numeric_only, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10697, in mean
return self._stat_function(
File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 10649, in _stat_function
return self._reduce(
File "/usr/local/lib/python3.10/site-packages/pandas/core/frame.py", line 10018, in _reduce
stacklevel=find_stack_level(),
File "/usr/local/lib/python3.10/site-packages/pandas/util/_exceptions.py", line 32, in find_stack_level
stack = inspect.stack()
File "/usr/local/lib/python3.10/inspect.py", line 1678, in stack
return getouterframes(sys._getframe(1), context)
File "/usr/local/lib/python3.10/inspect.py", line 1655, in getouterframes
frameinfo = (frame,) + getframeinfo(frame, context)
File "/usr/local/lib/python3.10/inspect.py", line 1627, in getframeinfo
start = lineno - 1 - context//2
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' [while running 'ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-f9a7f483-a708-404c-bfa7-41d2db50d9f3/output.csv/[ComputedExpression[apply_DataFrame_140679440005824]]:140679440571056/FlatMap(evaluate)/FlatMap(evaluate)-ptransform-81']
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:00:07.399Z: JOB_MESSAGE_DETAILED: Cleaning up.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:00:07.730Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:00:07.755Z: JOB_MESSAGE_BASIC: Stopping worker pool...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:02:25.918Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:02:26.008Z: JOB_MESSAGE_BASIC: Worker pool stopped.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243 2022-10-18T20:02:26.043Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job 2022-10-18_12_53_37-8879752588403415834 is in state JOB_STATE_FAILED
[31m[1mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1640 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2022-10-18_12_53_37-8879752588403415834?project=<ProjectId>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
../../build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py:17
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/operations_v1/abstract_operations_client.py>:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils import util
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:659: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1987: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1126: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.project = self.project or p.options.view_as(GoogleCloudOptions).project
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py: 2 warnings
apache_beam/examples/cookbook/filters_test.py: 2 warnings
apache_beam/examples/complete/game/leader_board_it_test.py: 2 warnings
apache_beam/examples/complete/game/game_stats_it_test.py: 3 warnings
apache_beam/examples/dataframe/flight_delays_it_test.py: 3 warnings
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py>:147: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = ssl.SSLContext(DEFAULT_TLS_VERSION)
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it
apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py310-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it
FAILED apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
[31m====== [31m[1m2 failed[0m, [32m15 passed[0m, [33m4 skipped[0m, [33m99 warnings[0m[31m in 1366.85s (0:22:46)[0m[31m =======[0m
> Task :sdks:python:test-suites:dataflow:py310:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py310:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 28m 34s
15 actionable tasks: 15 executed
Publishing build scan...
https://gradle.com/s/pq7yzjyapbdds
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1013
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1013/display/redirect?page=changes>
Changes:
[Alexey Romanenko] [TPC-DS] Use "nonpartitioned" input for Jenkins jobs
[noreply] [GitHub Actions] - Run RC Validations Workflow (#23531)
[noreply] Add workflow to update milestone on issue close (#23629)
[noreply] add website page about data processing for ML (#23552)
------------------------------------------
[...truncated 174.27 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7f866d732a00>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7f86455d5040>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp81y60g5t/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f863635a550> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f863635ad30> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f866ddd5eb0>
http = <httplib2.Http object at 0x7f866ddd3580>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f86372035e0>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F9de38bd9-3148-4484-b337-ff225198c3fb%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdvKvsSygzinK1PKTtfC7uLMVf0aN43WRwa5F0nk49SJzchB8Dsnne8hILbkP8uxlw0kiruNJmiik-_o2ZffO5u7Mg', 'date': 'Tue, 18 Oct 2022 14:42:47 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/9de38bd9-3148-4484-b337-ff225198c3fb/input.txt
[31m[1mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/9de38bd9-3148-4484-b337-ff225198c3fb/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F9de38bd9-3148-4484-b337-ff225198c3fb%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdvKvsSygzinK1PKTtfC7uLMVf0aN43WRwa5F0nk49SJzchB8Dsnne8hILbkP8uxlw0kiruNJmiik-_o2ZffO5u7Mg', 'date': 'Tue, 18 Oct 2022 14:42:47 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4871.66s (1:21:11)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 47m 25s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/5vcp7gq5gizeg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1012
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1012/display/redirect?page=changes>
Changes:
[noreply] Fix python log_level_overrides cannot be used on flink and other
[noreply] Better error for disabling runner v2 with cross language pipelines.
[noreply] Update scopes to match the existing set and update test to clear
------------------------------------------
[...truncated 174.22 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7f3df280e250>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7f3dc66e8040>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp1s08o42l/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:484 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f3da345e550> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f3da345ed30> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7f3df28d7ee0>
http = <httplib2.Http object at 0x7f3df2eeed90>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7f3df2eeed00>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F12c5e1a1-6fd7-40a3-b429-11f6ef6912b7%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdt63FdVsNADID97JknjSQ3-hTj5iBp0OMMBrWINXBP5b9I8mJaaIGFEMuX2a4OLbCDRvPpF1Xy4nDozZfWle5Fpd3GZYAgk', 'date': 'Tue, 18 Oct 2022 08:45:14 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/12c5e1a1-6fd7-40a3-b429-11f6ef6912b7/input.txt
[31m[1mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/12c5e1a1-6fd7-40a3-b429-11f6ef6912b7/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2F12c5e1a1-6fd7-40a3-b429-11f6ef6912b7%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdt63FdVsNADID97JknjSQ3-hTj5iBp0OMMBrWINXBP5b9I8mJaaIGFEMuX2a4OLbCDRvPpF1Xy4nDozZfWle5Fpd3GZYAgk', 'date': 'Tue, 18 Oct 2022 08:45:14 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4877.65s (1:21:17)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 49m 52s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/73rgwnljeuuom
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Dataflow #1011
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/1011/display/redirect?page=changes>
Changes:
[noreply] Fixes #23627: Speed up website checks (#23673)
[noreply] Suppress a FloatingPointLiteralPrecision error (#23667)
[noreply] Improved test coverage and fix the implementation of Inject and CoGBK
------------------------------------------
[...truncated 174.22 KB...]
client_secret = 'x_Tw5K8nnjoRAqULM9PFAC2b'
user_agent = 'x_Tw5K8nnjoRAqULM9PFAC2b'
credentials_filename = '/home/jenkins/.apitools.token', api_key = None
client = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3 object at 0x7fc890de1220>
oauth2client_args = None, kwds = {}
client_info = {'client_id': '1042881264118.apps.googleusercontent.com', 'client_secret': 'x_Tw5K8nnjoRAqULM9PFAC2b', 'scope': 'https...is.com/auth/compute.readonly https://www.googleapis.com/auth/userinfo.email', 'user_agent': 'x_Tw5K8nnjoRAqULM9PFAC2b'}
method = <function _GetApplicationDefaultCredentials at 0x7fc868c40ee0>
credentials = None
def GetCredentials(package_name, scopes, client_id, client_secret, user_agent,
credentials_filename=None,
api_key=None, # pylint: disable=unused-argument
client=None, # pylint: disable=unused-argument
oauth2client_args=None,
**kwds):
"""Attempt to get credentials, using an oauth dance as the last resort."""
scopes = util.NormalizeScopes(scopes)
client_info = {
'client_id': client_id,
'client_secret': client_secret,
'scope': ' '.join(sorted(scopes)),
'user_agent': user_agent or '%s-generated/0.1' % package_name,
}
for method in _CREDENTIALS_METHODS:
credentials = method(client_info, **kwds)
if credentials is not None:
return credentials
credentials_filename = credentials_filename or os.path.expanduser(
'~/.apitools.token')
credentials = CredentialsFromFile(credentials_filename, client_info,
oauth2client_args=oauth2client_args)
if credentials is not None:
return credentials
> raise exceptions.CredentialsError('Could not create valid credentials')
[1m[31mE apitools.base.py.exceptions.CredentialsError: Could not create valid credentials[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/credentials_lib.py[0m:157: CredentialsError
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/bin/python3.9',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmp4eqf_bl3/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
[32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:480 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.43.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914
[32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220914" for Docker environment
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7fc849993430> ====================
[32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7fc849993c10> ====================
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[33mWARNING [0m oauth2client.contrib.multiprocess_file_storage:multiprocess_file_storage.py:141 Credentials file could not be loaded, will ignore and overwrite.
[31m[1m_______________ CodersIT.test_coders_output_files_on_small_input _______________[0m
self = <apache_beam.examples.cookbook.coders_it_test.CodersIT testMethod=test_coders_output_files_on_small_input>
@pytest.mark.no_xdist
@pytest.mark.examples_postcommit
def test_coders_output_files_on_small_input(self):
test_pipeline = TestPipeline(is_integration_test=True)
# Setup the files with expected content.
OUTPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output'
output = '/'.join([OUTPUT_FILE_DIR, str(uuid.uuid4()), 'result'])
INPUT_FILE_DIR = \
'gs://temp-storage-for-end-to-end-tests/py-it-cloud/input'
input = '/'.join([INPUT_FILE_DIR, str(uuid.uuid4()), 'input.txt'])
> create_content_input_file(
input, '\n'.join(map(json.dumps, self.SAMPLE_RECORDS)))
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:90:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/examples/cookbook/coders_it_test.py[0m:51: in create_content_input_file
f.write(str.encode(contents, 'utf-8'))
[1m[31mapache_beam/io/filesystemio.py[0m:215: in close
self._uploader.finish()
[1m[31mapache_beam/io/gcp/gcsio.py[0m:790: in finish
raise self._upload_thread.last_error # pylint: disable=raising-bad-type
[1m[31mapache_beam/io/gcp/gcsio.py[0m:763: in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
[1m[31mapache_beam/io/gcp/internal/clients/storage/storage_v1_client.py[0m:1152: in Insert
return self._RunMethod(
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py[0m:714: in _RunMethod
http_response = upload.InitializeUpload(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Upload (uninitialized)
http_request = <apitools.base.py.http_wrapper.Request object at 0x7fc895c15910>
http = <httplib2.Http object at 0x7fc84a84a100>
client = <apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1 object at 0x7fc89145c370>
def InitializeUpload(self, http_request, http=None, client=None):
"""Initialize this upload from the given http_request."""
if self.strategy is None:
raise exceptions.UserError(
'No upload strategy set; did you call ConfigureRequest?')
if http is None and client is None:
raise exceptions.UserError('Must provide client or http.')
if self.strategy != RESUMABLE_UPLOAD:
return
http = http or client.http
if client is not None:
http_request.url = client.FinalizeTransferUrl(http_request.url)
self.EnsureUninitialized()
http_response = http_wrapper.MakeRequest(http, http_request,
retries=self.num_retries)
if http_response.status_code != http_client.OK:
> raise exceptions.HttpError.FromResponse(http_response)
[1m[31mE apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fca5c6192-9dbf-4296-9935-6f8fc29557b4%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdtxPi0VR07bC5ezg4ZMo1Atyskd2dLOHvbeJ8WnpwyG4uWwjrpoIk0iTCYnvLvVeRSyfpr-_IR1pVj4fG9Gpqp7uoRkg5Mz', 'date': 'Tue, 18 Oct 2022 02:37:28 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{[0m
[1m[31mE "error": {[0m
[1m[31mE "code": 401,[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "errors": [[0m
[1m[31mE {[0m
[1m[31mE "message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",[0m
[1m[31mE "domain": "global",[0m
[1m[31mE "reason": "required",[0m
[1m[31mE "locationType": "header",[0m
[1m[31mE "location": "Authorization"[0m
[1m[31mE }[0m
[1m[31mE ][0m
[1m[31mE }[0m
[1m[31mE }[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py[0m:896: HttpUnauthorizedError
------------------------------ Captured log call -------------------------------
[32mINFO [0m root:coders_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/ca5c6192-9dbf-4296-9935-6f8fc29557b4/input.txt
[1m[31mERROR [0m apache_beam.io.gcp.gcsio:gcsio.py:767 Error in _start_upload while inserting file gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/ca5c6192-9dbf-4296-9935-6f8fc29557b4/input.txt: Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py",> line 763, in _start_upload
self._client.objects.Insert(self._insert_request, upload=self._upload)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",> line 1152, in Insert
return self._RunMethod(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/base_api.py",> line 714, in _RunMethod
http_response = upload.InitializeUpload(
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/apitools/base/py/transfer.py",> line 896, in InitializeUpload
raise exceptions.HttpError.FromResponse(http_response)
apitools.base.py.exceptions.HttpUnauthorizedError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/temp-storage-for-end-to-end-tests/o?alt=json&name=py-it-cloud%2Finput%2Fca5c6192-9dbf-4296-9935-6f8fc29557b4%2Finput.txt&uploadType=resumable>: response: <{'x-guploader-uploadid': 'ADPycdtxPi0VR07bC5ezg4ZMo1Atyskd2dLOHvbeJ8WnpwyG4uWwjrpoIk0iTCYnvLvVeRSyfpr-_IR1pVj4fG9Gpqp7uoRkg5Mz', 'date': 'Tue, 18 Oct 2022 02:37:28 GMT', 'vary': 'Origin, X-Origin', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '438', 'server': 'UploadServer', 'content-type': 'text/html; charset=UTF-8', 'status': '401'}>, content <{
"error": {
"code": 401,
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"errors": [
{
"message": "Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.",
"domain": "global",
"reason": "required",
"locationType": "header",
"location": "Authorization"
}
]
}
}
>
[33m=============================== warnings summary ===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/typehints/pandas_type_compatibility_test.py:66
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
}).set_index(pd.Int64Index(range(123, 223), name='an_index')),
apache_beam/typehints/pandas_type_compatibility_test.py:89
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(123, 223), name='an_index'),
apache_beam/typehints/pandas_type_compatibility_test.py:90
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
pd.Int64Index(range(475, 575), name='another_index'),
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1991: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1997: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1133: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1135: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2484: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2486: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2510: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-no-xdist.xml> -
=========================== short test summary info ============================
FAILED apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input
FAILED apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file
FAILED apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
FAILED apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
FAILED apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
FAILED apache_beam/examples/cookbook/coders_it_test.py::CodersIT::test_coders_output_files_on_small_input
ERROR apache_beam/examples/complete/game/user_score_it_test.py::UserScoreIT::test_userscore_output_checksum_on_small_input
ERROR apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
[31m= [31m[1m6 failed[0m, [33m3 skipped[0m, [33m6649 deselected[0m, [33m16 warnings[0m, [31m[1m2 errors[0m[31m in 4872.23s (1:21:12)[0m[31m =[0m
> Task :sdks:python:test-suites:dataflow:py39:examples FAILED
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 203
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 42m 5s
15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/losvbzftmzkoa
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org