You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2022/02/08 00:28:36 UTC
Build failed in Jenkins: beam_PostCommit_Python_Examples_Spark #13
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/13/display/redirect>
Changes:
------------------------------------------
[...truncated 3.14 MB...]
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:24 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:24 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 0
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:164 Creating insecure control channel for localhost:35379.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:172 Control channel established.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:215 Initializing SDKHarness with unbounded number of workers.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:24 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 131-1'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:24 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: getProcessBundleDescriptor request with id 131-3'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:24 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: getProcessBundleDescriptor request with id 131-2'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:807 Creating insecure state channel for localhost:41651.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:24 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: getProcessBundleDescriptor request with id 131-4'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:814 State channel established.
INFO apache_beam.runners.worker.data_plane:data_plane.py:750 Creating client data channel for localhost:44731
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:24 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.'
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4598189188425841853_95054b97-f2f4-472c-bb36-5f1dad6918ff to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4598189188425841853_95054b97-f2f4-472c-bb36-5f1dad6918ff', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4802339249571603211_4cc863c7-3d2a-47f7-80fc-0d3a2d493571 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4802339249571603211_4cc863c7-3d2a-47f7-80fc-0d3a2d493571', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/474657147301083487_2907f928-2f44-4777-ba76-cb33c56ca487 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/474657147301083487_2907f928-2f44-4777-ba76-cb33c56ca487', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4598189188425841853_49aed087-c578-4287-8d1d-74b87f559c17 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4598189188425841853_49aed087-c578-4287-8d1d-74b87f559c17', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4802339249571603211_6ca6b539-164f-4e26-9123-f0898fa5a25e to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4802339249571603211_6ca6b539-164f-4e26-9123-f0898fa5a25e', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/474657147301083487_20705ddd-98b3-43c8-b793-a5e8771d15c8 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/474657147301083487_20705ddd-98b3-43c8-b793-a5e8771d15c8', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4598189188425841853_c2773fd8-af06-4897-a760-293be2009761 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4598189188425841853_c2773fd8-af06-4897-a760-293be2009761', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/474657147301083487_7434ccee-f8ac-4a18-b84b-0ceeb080939c to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/474657147301083487_7434ccee-f8ac-4a18-b84b-0ceeb080939c', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4802339249571603211_914d398d-b061-4bcc-95b0-0b2cc8eb3cf4 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4802339249571603211_914d398d-b061-4bcc-95b0-0b2cc8eb3cf4', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4598189188425841853_9b3a6db1-3937-43df-b05e-f80e043a9ffe to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4598189188425841853_9b3a6db1-3937-43df-b05e-f80e043a9ffe', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4802339249571603211_7eaa0c09-1d7b-4425-a233-3ae0902548e1 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/4802339249571603211_7eaa0c09-1d7b-4425-a233-3ae0902548e1', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:637 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/474657147301083487_b7c26080-1367-4908-9049-0cf077375fbe to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-4126c3a1-820c-48cf-b212-006768277b48 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempa07891ef-a2b4-4927-9595-7ad4d060d801/474657147301083487_b7c26080-1367-4908-9049-0cf077375fbe', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:661 Checking orphaned temporary files for destination None and window [1356307200.0, 1356393600.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.fileio:fileio.py:661 Checking orphaned temporary files for destination None and window [1356393600.0, 1356480000.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.fileio:fileio.py:661 Checking orphaned temporary files for destination None and window [1356220800.0, 1356307200.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 0 files in 0.05951428413391113 seconds.
INFO apache_beam.io.fileio:fileio.py:676 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 0 files in 0.08423972129821777 seconds.
INFO apache_beam.io.fileio:fileio.py:676 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 0 files in 0.07470154762268066 seconds.
INFO apache_beam.io.fileio:fileio.py:676 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:25 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:external:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'payload: "\\n\\021\\n\\017localhost:37229"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bytes:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:string_utf8:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:kv:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bool:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:varint:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:double:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:iterable:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:timer:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:interval_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:length_prefix:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:global_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:param_windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:state_backed_iterable:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:custom_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:row:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:sharded_key:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:progress_reporting:v0"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:harness_monitoring_infos:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:worker_status:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:combinefn:packed_python:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:version:sdk_base:apache/beam_python3.8_sdk:2.37.0.dev"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:to_string:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b''
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:25 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:244 No more requests from control plane
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:245 SDK Harness waiting for in-flight requests to complete
INFO apache_beam.runners.worker.data_plane:data_plane.py:782 Closing all cached grpc data channels.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:826 Closing all cached gRPC state handlers.
ERROR apache_beam.runners.worker.data_plane:data_plane.py:641 Failed to read inputs in the data plane.
Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644280105.790193046","description":"Error received from peer ipv6:[::1]:44731","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:257 Done consuming work.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 00:28:25 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job BeamApp-jenkins-0208002751-e9a174df_bbfca6e2-31cc-45e4-b134-a2c95b384f9d finished.'
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:574 Job state changed to DONE
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 4 files in 0.03937339782714844 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 12 files in 0.04899930953979492 seconds.
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2463: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
pipeline_options=pcoll.pipeline.options,
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:629: FutureWarning: WriteToFiles is experimental.
return pcoll | fileio.WriteToFiles(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py38.xml> -
= 1 failed, 21 passed, 1 skipped, 5178 deselected, 82 warnings in 293.07 seconds =
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 248
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py38:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 8m 4s
79 actionable tasks: 52 executed, 25 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/srllvqb6tahnu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_Python_Examples_Spark #22
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/22/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Spark #21
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/21/display/redirect?page=changes>
Changes:
[david.prieto.rivera] Missing contribution
[noreply] Merge pull request #15848 from [BEAM-13835] An any-type implementation
[Valentyn Tymofieiev] [BEAM-12920] Assume that bare generators types define simple generators.
[noreply] [release-2.36.0][website] Fix github release notes script, header for
[noreply] Use shell to run python for setupVirtualenv (#16796)
[Daniel Oliveira] [BEAM-13830] Properly shut down Debezium expansion service in IT script.
[noreply] Merge pull request #16659 from [BEAM-13774][Playground] Add user to
[noreply] [BEAM-13776][Playground] (#16731)
------------------------------------------
[...truncated 3.17 MB...]
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.03903937339782715 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.0432584285736084 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/10 00:22:53 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:external:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'payload: "\\n\\021\\n\\017localhost:37603"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bytes:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:string_utf8:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:kv:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bool:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:244 No more requests from control plane
ERROR apache_beam.runners.worker.data_plane:data_plane.py:641 Failed to read inputs in the data plane.
Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644452573.798698010","description":"Error received from peer ipv6:[::1]:32955","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:varint:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:245 SDK Harness waiting for in-flight requests to complete
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:double:v1"'
INFO apache_beam.runners.worker.data_plane:data_plane.py:782 Closing all cached grpc data channels.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:iterable:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:826 Closing all cached gRPC state handlers.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:timer:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:interval_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:length_prefix:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:257 Done consuming work.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:global_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:param_windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:state_backed_iterable:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:custom_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:row:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:sharded_key:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:progress_reporting:v0"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:harness_monitoring_infos:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:worker_status:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:combinefn:packed_python:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:version:sdk_base:apache/beam_python3.7_sdk:2.37.0.dev"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:to_string:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b''
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/10 00:22:53 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/10 00:22:53 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job BeamApp-jenkins-0210002231-6ac583c5_b9b7e459-44ba-488a-bab3-9b01ba4d1160 finished.'
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:576 Job state changed to DONE
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.03881978988647461 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.0488133430480957 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.03709292411804199 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 12 files in 0.04759550094604492 seconds.
PASSED [ 95%]
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
-------------------------------- live log call ---------------------------------
INFO root:pipeline.py:188 Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_column_DataFrame_140663561685904], ComputedExpression[set_index_DataFrame_140663561683472], ComputedExpression[pre_combine_sum_DataFrame_140665108525392]]:140663557980688]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_140665890329744]}, partitioning=Arbitrary, ops=[ComputedExpression[set_column_DataFrame_140663561685904], ComputedExpression[set_index_DataFrame_140663561683472], ComputedExpression[pre_combine_sum_DataFrame_140665108525392]], outputs={PlaceholderExpression[placeholder_DataFrame_140665890329744], ComputedExpression[pre_combine_sum_DataFrame_140665108525392]}]
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[post_combine_sum_DataFrame_140663561649872]]:140663561787664]> for Stage[inputs={ComputedExpression[pre_combine_sum_DataFrame_140665108525392]}, partitioning=Index, ops=[ComputedExpression[post_combine_sum_DataFrame_140663561649872]], outputs={ComputedExpression[post_combine_sum_DataFrame_140663561649872]}]
INFO apache_beam.io.fileio:fileio.py:555 Added temporary directory /tmp/.tempdb1ec988-867d-4277-8771-5d57cb76f07d
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.7_sdk:2.37.0.dev
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function annotate_downstream_side_inputs at 0x7fef2ab4c7a0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function fix_side_input_pcoll_coders at 0x7fef2ab4c8c0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function pack_combiners at 0x7fef2ab4cdd0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function lift_combiners at 0x7fef2ab4ce60> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_sdf at 0x7fef2ab44050> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_gbk at 0x7fef2ab440e0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sink_flattens at 0x7fef2ab44200> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function greedily_fuse at 0x7fef2ab44290> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function read_to_impulse at 0x7fef2ab44320> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function impulse_to_input at 0x7fef2ab443b0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sort_stages at 0x7fef2ab445f0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function setup_timer_mapping at 0x7fef2ab44560> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function populate_data_channel_coders at 0x7fef2ab44680> ====================
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 100
INFO apache_beam.runners.portability.fn_api_runner.worker_handlers:worker_handlers.py:894 Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7fef262b01d0> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((ref_AppliedPTransform_Read-Read-Impulse_4)+(ref_AppliedPTransform_Read-Read-Map-lambda-at-iobase-py-898-_5))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((((ref_PCollection_PCollection_2_split/Read)+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Split_8))+(ref_AppliedPTransform_ToRows_9))+(ref_AppliedPTransform_BatchElements-words-BatchElements-ParDo-_GlobalWindowsBatchingDoFn-_12))+(ref_AppliedPTransform_BatchElements-words-Map-lambda-at-schemas-py-140-_13))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-set_column_DataFr_16))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-set_column_DataFr_18))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-post_combine_sum__21))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-post_combine_sum__22))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-post_combine_sum__23))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-post_combine_sum__26))+(ToPCollection(df) - /tmp/tmp12ydq6_n.result/[ComputedExpression[post_combine_sum_DataFrame_140663561649872]]:140663561787664/CoGroupByKey/CoGroupByKeyImpl/Flatten/Transcode/0))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-post_combine_sum__27))+(ToPCollection(df) - /tmp/tmp12ydq6_n.result/[ComputedExpression[post_combine_sum_DataFrame_140663561649872]]:140663561787664/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((ToPCollection(df) - /tmp/tmp12ydq6_n.result/[ComputedExpression[post_combine_sum_DataFrame_140663561649872]]:140663561787664/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-post_combine_sum__29))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-post_combine_sum__30))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-post_combine_sum__31))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp12ydq6_n-result-ComputedExpression-post_combine_sum__33))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp12ydq6_n-result-WriteToFiles-ParDo-_WriteUnshardedRe_37))+(ref_AppliedPTransform_Unbatch-post_combine_sum_DataFrame_140663561649872-with-indexes-ParDo-_Unbatch_46))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp12ydq6_n-result-WriteToFiles-ParDo-_AppendShardedDes_38))+(WriteToPandas(df) - /tmp/tmp12ydq6_n.result/WriteToFiles/Flatten/Write/0))+(WriteToPandas(df) - /tmp/tmp12ydq6_n.result/WriteToFiles/GroupRecordsByDestinationAndShard/Write))+(ref_AppliedPTransform_Filter-lambda-at-wordcount-py-80-_47))+(ref_AppliedPTransform_Map-lambda-at-wordcount-py-81-_48))+(ref_AppliedPTransform_Map-print-_49)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp12ydq6_n.result/WriteToFiles/GroupRecordsByDestinationAndShard/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp12ydq6_n-result-WriteToFiles-ParDo-_WriteShardedReco_40))+(WriteToPandas(df) - /tmp/tmp12ydq6_n.result/WriteToFiles/Flatten/Write/1)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp12ydq6_n.result/WriteToFiles/Flatten/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp12ydq6_n-result-WriteToFiles-Map-lambda-at-fileio-py_42))+(WriteToPandas(df) - /tmp/tmp12ydq6_n.result/WriteToFiles/GroupTempFilesByDestination/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running (WriteToPandas(df) - /tmp/tmp12ydq6_n.result/WriteToFiles/GroupTempFilesByDestination/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp12ydq6_n-result-WriteToFiles-ParDo-_MoveTempFilesInt_44)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file /tmp/.tempdb1ec988-867d-4277-8771-5d57cb76f07d/1356248840311852570_990cebc3-1dbe-4f6f-b204-6207c1b68045 to dir: /tmp as tmp12ydq6_n.result-00000-of-00001. Res: FileResult(file_name='/tmp/.tempdb1ec988-867d-4277-8771-5d57cb76f07d/1356248840311852570_990cebc3-1dbe-4f6f-b204-6207c1b68045', shard_index=-1, total_shards=0, window=GlobalWindow, pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window GlobalWindow
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
PASSED [100%]
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2470: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:632: FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py37.xml> -
===== 22 passed, 1 skipped, 5184 deselected, 39 warnings in 265.30 seconds =====
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644452530.059189296","description":"Error received from peer ipv6:[::1]:44433","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644452570.680984687","description":"Error received from peer ipv6:[::1]:39431","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 248
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py38:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 7m 21s
79 actionable tasks: 52 executed, 25 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/nbj56nmqnvfds
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Spark #20
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/20/display/redirect?page=changes>
Changes:
[noreply] [BEAM-13803] Add support for native iterable side inputs to the Go SDK
[noreply] [BEAM-11095] Better error handling for illegal emit functions (#16776)
[noreply] Merge pull request #16613 from Supporting JdbcIO driver in classpath for
------------------------------------------
[...truncated 3.16 MB...]
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:sharded_key:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:progress_reporting:v0"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:harness_monitoring_infos:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:worker_status:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:combinefn:packed_python:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:version:sdk_base:apache/beam_python3.7_sdk:2.37.0.dev"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:to_string:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b''
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/09 18:08:27 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/09 18:08:27 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job BeamApp-jenkins-0209180805-7827896b_71c1851e-3472-42b6-926d-220c68b5a109 finished.'
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:576 Job state changed to DONE
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.03394460678100586 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.04135465621948242 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.030356645584106445 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 12 files in 0.03612780570983887 seconds.
PASSED [ 95%]
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
-------------------------------- live log call ---------------------------------
INFO root:pipeline.py:188 Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_column_DataFrame_139880314567440], ComputedExpression[set_index_DataFrame_139880314568336], ComputedExpression[pre_combine_sum_DataFrame_139882074142672]]:139881651795024]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_139882074919440]}, partitioning=Arbitrary, ops=[ComputedExpression[set_column_DataFrame_139880314567440], ComputedExpression[set_index_DataFrame_139880314568336], ComputedExpression[pre_combine_sum_DataFrame_139882074142672]], outputs={ComputedExpression[pre_combine_sum_DataFrame_139882074142672], PlaceholderExpression[placeholder_DataFrame_139882074919440]}]
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[post_combine_sum_DataFrame_139882072097104]]:139882075349584]> for Stage[inputs={ComputedExpression[pre_combine_sum_DataFrame_139882074142672]}, partitioning=Index, ops=[ComputedExpression[post_combine_sum_DataFrame_139882072097104]], outputs={ComputedExpression[post_combine_sum_DataFrame_139882072097104]}]
INFO apache_beam.io.fileio:fileio.py:555 Added temporary directory /tmp/.temp16092784-763d-4fae-bb6c-ba0b5578e0b1
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.7_sdk:2.37.0.dev
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function annotate_downstream_side_inputs at 0x7f38c16bf950> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function fix_side_input_pcoll_coders at 0x7f38c16bfa70> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function pack_combiners at 0x7f38c16bff80> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function lift_combiners at 0x7f38c16be050> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_sdf at 0x7f38c16be200> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_gbk at 0x7f38c16be290> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sink_flattens at 0x7f38c16be3b0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function greedily_fuse at 0x7f38c16be440> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function read_to_impulse at 0x7f38c16be4d0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function impulse_to_input at 0x7f38c16be560> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sort_stages at 0x7f38c16be7a0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function setup_timer_mapping at 0x7f38c16be710> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function populate_data_channel_coders at 0x7f38c16be830> ====================
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 100
INFO apache_beam.runners.portability.fn_api_runner.worker_handlers:worker_handlers.py:894 Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7f38d504d710> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((ref_AppliedPTransform_Read-Read-Impulse_4)+(ref_AppliedPTransform_Read-Read-Map-lambda-at-iobase-py-898-_5))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((((ref_PCollection_PCollection_2_split/Read)+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Split_8))+(ref_AppliedPTransform_ToRows_9))+(ref_AppliedPTransform_BatchElements-words-BatchElements-ParDo-_GlobalWindowsBatchingDoFn-_12))+(ref_AppliedPTransform_BatchElements-words-Map-lambda-at-schemas-py-140-_13))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-set_column_DataFr_16))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-set_column_DataFr_18))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-post_combine_sum__21))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-post_combine_sum__22))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-post_combine_sum__23))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-post_combine_sum__26))+(ToPCollection(df) - /tmp/tmpinctmc7r.result/[ComputedExpression[post_combine_sum_DataFrame_139882072097104]]:139882075349584/CoGroupByKey/CoGroupByKeyImpl/Flatten/Transcode/0))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-post_combine_sum__27))+(ToPCollection(df) - /tmp/tmpinctmc7r.result/[ComputedExpression[post_combine_sum_DataFrame_139882072097104]]:139882075349584/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((ToPCollection(df) - /tmp/tmpinctmc7r.result/[ComputedExpression[post_combine_sum_DataFrame_139882072097104]]:139882075349584/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-post_combine_sum__29))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-post_combine_sum__30))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-post_combine_sum__31))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpinctmc7r-result-ComputedExpression-post_combine_sum__33))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpinctmc7r-result-WriteToFiles-ParDo-_WriteUnshardedRe_37))+(ref_AppliedPTransform_Unbatch-post_combine_sum_DataFrame_139882072097104-with-indexes-ParDo-_Unbatch_46))+(WriteToPandas(df) - /tmp/tmpinctmc7r.result/WriteToFiles/Flatten/Write/0))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpinctmc7r-result-WriteToFiles-ParDo-_AppendShardedDes_38))+(WriteToPandas(df) - /tmp/tmpinctmc7r.result/WriteToFiles/GroupRecordsByDestinationAndShard/Write))+(ref_AppliedPTransform_Filter-lambda-at-wordcount-py-80-_47))+(ref_AppliedPTransform_Map-lambda-at-wordcount-py-81-_48))+(ref_AppliedPTransform_Map-print-_49)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmpinctmc7r.result/WriteToFiles/GroupRecordsByDestinationAndShard/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpinctmc7r-result-WriteToFiles-ParDo-_WriteShardedReco_40))+(WriteToPandas(df) - /tmp/tmpinctmc7r.result/WriteToFiles/Flatten/Write/1)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmpinctmc7r.result/WriteToFiles/Flatten/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpinctmc7r-result-WriteToFiles-Map-lambda-at-fileio-py_42))+(WriteToPandas(df) - /tmp/tmpinctmc7r.result/WriteToFiles/GroupTempFilesByDestination/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running (WriteToPandas(df) - /tmp/tmpinctmc7r.result/WriteToFiles/GroupTempFilesByDestination/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpinctmc7r-result-WriteToFiles-ParDo-_MoveTempFilesInt_44)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file /tmp/.temp16092784-763d-4fae-bb6c-ba0b5578e0b1/8237360927517950451_99d850dc-c7e1-4ae8-8528-89694de60bda to dir: /tmp as tmpinctmc7r.result-00000-of-00001. Res: FileResult(file_name='/tmp/.temp16092784-763d-4fae-bb6c-ba0b5578e0b1/8237360927517950451_99d850dc-c7e1-4ae8-8528-89694de60bda', shard_index=-1, total_shards=0, window=GlobalWindow, pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window GlobalWindow
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
PASSED [100%]
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2470: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:632: FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py37.xml> -
===== 22 passed, 1 skipped, 5184 deselected, 39 warnings in 291.02 seconds =====
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644430067.569271512","description":"Error received from peer ipv6:[::1]:44509","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644430104.676333419","description":"Error received from peer ipv6:[::1]:39535","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644430104.817925414","description":"Error received from peer ipv6:[::1]:45253","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644430105.026488156","description":"Error received from peer ipv6:[::1]:43655","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 248
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py38:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 6m 32s
79 actionable tasks: 50 executed, 27 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/o4coiapax6hgi
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Spark #19
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/19/display/redirect?page=changes>
Changes:
[mmack] [adhoc] Remove remaining usage of Powermock from aws2.
------------------------------------------
[...truncated 3.06 MB...]
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/3775490355647198002_4c4d1b13-528e-4440-a261-0499b71ee16d to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/3775490355647198002_4c4d1b13-528e-4440-a261-0499b71ee16d', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4878869925992615118_97d096f5-5ca6-4d72-b942-a03854727859 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4878869925992615118_97d096f5-5ca6-4d72-b942-a03854727859', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4528305008856648910_8bdd2692-ffa9-417e-b480-e0add35796b0 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4528305008856648910_8bdd2692-ffa9-417e-b480-e0add35796b0', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4878869925992615118_eca7ffb5-3b86-420e-80ce-c3404d0e6f73 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4878869925992615118_eca7ffb5-3b86-420e-80ce-c3404d0e6f73', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4528305008856648910_882d4e08-3b4e-48d7-8548-c06c37a62801 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4528305008856648910_882d4e08-3b4e-48d7-8548-c06c37a62801', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/3775490355647198002_f963641a-642a-4ef5-ba74-480f9d6a6df5 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/3775490355647198002_f963641a-642a-4ef5-ba74-480f9d6a6df5', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4528305008856648910_b54c8bac-4c5d-4006-905d-3a6485733efe to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4528305008856648910_b54c8bac-4c5d-4006-905d-3a6485733efe', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4878869925992615118_195e36c2-ab92-4d6c-901a-6df1b63ac468 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4878869925992615118_195e36c2-ab92-4d6c-901a-6df1b63ac468', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/3775490355647198002_ec4a526a-d51e-47de-b834-2a88f0313731 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/3775490355647198002_ec4a526a-d51e-47de-b834-2a88f0313731', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4878869925992615118_5deb39ba-ebd1-47bd-8a3a-5b69301ef6e8 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4878869925992615118_5deb39ba-ebd1-47bd-8a3a-5b69301ef6e8', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4528305008856648910_2eb81e91-31d4-4033-9439-91a34efcc1ca to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/4528305008856648910_2eb81e91-31d4-4033-9439-91a34efcc1ca', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/3775490355647198002_28a0fa74-9997-465c-9fe5-4df70d0ef1eb to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-5c9b279b-a072-4957-8e25-170da9047d15 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp7bb8aec0-0501-4ad8-a1d6-fff7d0df67d7/3775490355647198002_28a0fa74-9997-465c-9fe5-4df70d0ef1eb', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356393600.0, 1356480000.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356220800.0, 1356307200.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356307200.0, 1356393600.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.05432534217834473 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.03295731544494629 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.04510927200317383 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/09 12:10:55 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:external:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'payload: "\\n\\021\\n\\017localhost:41485"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bytes:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:string_utf8:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:kv:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:244 No more requests from control plane
ERROR apache_beam.runners.worker.data_plane:data_plane.py:641 Failed to read inputs in the data plane.
Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644408655.724116386","description":"Error received from peer ipv6:[::1]:42737","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bool:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:245 SDK Harness waiting for in-flight requests to complete
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:varint:v1"'
INFO apache_beam.runners.worker.data_plane:data_plane.py:782 Closing all cached grpc data channels.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:double:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:826 Closing all cached gRPC state handlers.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:iterable:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:257 Done consuming work.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:timer:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:interval_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:length_prefix:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:global_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:param_windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:state_backed_iterable:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:custom_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:row:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:sharded_key:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:progress_reporting:v0"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:harness_monitoring_infos:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:worker_status:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:combinefn:packed_python:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:version:sdk_base:apache/beam_python3.7_sdk:2.37.0.dev"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:to_string:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b''
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/09 12:10:55 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/09 12:10:55 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job BeamApp-jenkins-0209121028-eb2b28c5_7cf4227b-4989-41ab-b299-e0c14892a245 finished.'
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:576 Job state changed to DONE
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.04907870292663574 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.05159783363342285 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.06420755386352539 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 12 files in 0.04588150978088379 seconds.
PASSED [ 95%]
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
-------------------------------- live log call ---------------------------------
INFO root:pipeline.py:188 Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_column_DataFrame_139796335161232], ComputedExpression[set_index_DataFrame_139796335233296], ComputedExpression[pre_combine_sum_DataFrame_139795884666960]]:139796333788240]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_139796337133072]}, partitioning=Arbitrary, ops=[ComputedExpression[set_column_DataFrame_139796335161232], ComputedExpression[set_index_DataFrame_139796335233296], ComputedExpression[pre_combine_sum_DataFrame_139795884666960]], outputs={PlaceholderExpression[placeholder_DataFrame_139796337133072], ComputedExpression[pre_combine_sum_DataFrame_139795884666960]}]
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[post_combine_sum_DataFrame_139795884669392]]:139796333786256]> for Stage[inputs={ComputedExpression[pre_combine_sum_DataFrame_139795884666960]}, partitioning=Index, ops=[ComputedExpression[post_combine_sum_DataFrame_139795884669392]], outputs={ComputedExpression[post_combine_sum_DataFrame_139795884669392]}]
INFO apache_beam.io.fileio:fileio.py:555 Added temporary directory /tmp/.tempc3b9956b-38f8-4b81-ac92-de458e1cec32
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.7_sdk:2.37.0.dev
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function annotate_downstream_side_inputs at 0x7f24cb1ff710> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function fix_side_input_pcoll_coders at 0x7f24cb1ff830> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function pack_combiners at 0x7f24cb1ffd40> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function lift_combiners at 0x7f24cb1ffdd0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_sdf at 0x7f24cb1fff80> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_gbk at 0x7f24cb1fd050> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sink_flattens at 0x7f24cb1fd170> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function greedily_fuse at 0x7f24cb1fd200> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function read_to_impulse at 0x7f24cb1fd290> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function impulse_to_input at 0x7f24cb1fd320> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sort_stages at 0x7f24cb1fd560> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function setup_timer_mapping at 0x7f24cb1fd4d0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function populate_data_channel_coders at 0x7f24cb1fd5f0> ====================
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 100
INFO apache_beam.runners.portability.fn_api_runner.worker_handlers:worker_handlers.py:894 Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7f24c41e0c10> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((ref_AppliedPTransform_Read-Read-Impulse_4)+(ref_AppliedPTransform_Read-Read-Map-lambda-at-iobase-py-898-_5))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((((ref_PCollection_PCollection_2_split/Read)+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Split_8))+(ref_AppliedPTransform_ToRows_9))+(ref_AppliedPTransform_BatchElements-words-BatchElements-ParDo-_GlobalWindowsBatchingDoFn-_12))+(ref_AppliedPTransform_BatchElements-words-Map-lambda-at-schemas-py-140-_13))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-set_column_DataFr_16))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-set_column_DataFr_18))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-post_combine_sum__21))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-post_combine_sum__22))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-post_combine_sum__23))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-post_combine_sum__26))+(ToPCollection(df) - /tmp/tmp0lakpaei.result/[ComputedExpression[post_combine_sum_DataFrame_139795884669392]]:139796333786256/CoGroupByKey/CoGroupByKeyImpl/Flatten/Transcode/0))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-post_combine_sum__27))+(ToPCollection(df) - /tmp/tmp0lakpaei.result/[ComputedExpression[post_combine_sum_DataFrame_139795884669392]]:139796333786256/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((ToPCollection(df) - /tmp/tmp0lakpaei.result/[ComputedExpression[post_combine_sum_DataFrame_139795884669392]]:139796333786256/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-post_combine_sum__29))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-post_combine_sum__30))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-post_combine_sum__31))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp0lakpaei-result-ComputedExpression-post_combine_sum__33))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp0lakpaei-result-WriteToFiles-ParDo-_WriteUnshardedRe_37))+(ref_AppliedPTransform_Unbatch-post_combine_sum_DataFrame_139795884669392-with-indexes-ParDo-_Unbatch_46))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp0lakpaei-result-WriteToFiles-ParDo-_AppendShardedDes_38))+(WriteToPandas(df) - /tmp/tmp0lakpaei.result/WriteToFiles/Flatten/Write/0))+(WriteToPandas(df) - /tmp/tmp0lakpaei.result/WriteToFiles/GroupRecordsByDestinationAndShard/Write))+(ref_AppliedPTransform_Filter-lambda-at-wordcount-py-80-_47))+(ref_AppliedPTransform_Map-lambda-at-wordcount-py-81-_48))+(ref_AppliedPTransform_Map-print-_49)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp0lakpaei.result/WriteToFiles/GroupRecordsByDestinationAndShard/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp0lakpaei-result-WriteToFiles-ParDo-_WriteShardedReco_40))+(WriteToPandas(df) - /tmp/tmp0lakpaei.result/WriteToFiles/Flatten/Write/1)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp0lakpaei.result/WriteToFiles/Flatten/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp0lakpaei-result-WriteToFiles-Map-lambda-at-fileio-py_42))+(WriteToPandas(df) - /tmp/tmp0lakpaei.result/WriteToFiles/GroupTempFilesByDestination/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running (WriteToPandas(df) - /tmp/tmp0lakpaei.result/WriteToFiles/GroupTempFilesByDestination/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp0lakpaei-result-WriteToFiles-ParDo-_MoveTempFilesInt_44)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file /tmp/.tempc3b9956b-38f8-4b81-ac92-de458e1cec32/2017027651076902129_54c03d5e-b346-4121-809d-25c024161899 to dir: /tmp as tmp0lakpaei.result-00000-of-00001. Res: FileResult(file_name='/tmp/.tempc3b9956b-38f8-4b81-ac92-de458e1cec32/2017027651076902129_54c03d5e-b346-4121-809d-25c024161899', shard_index=-1, total_shards=0, window=GlobalWindow, pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window GlobalWindow
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
PASSED [100%]
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2470: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:632: FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py37.xml> -
===== 22 passed, 1 skipped, 5181 deselected, 39 warnings in 275.65 seconds =====
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644408654.481516575","description":"Error received from peer ipv6:[::1]:34027","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 248
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py38:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 9m 1s
79 actionable tasks: 50 executed, 27 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/dtga4un7hjie2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Spark #18
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/18/display/redirect?page=changes>
Changes:
[Kyle Weaver] Update Dataflow Python dev container images.
[Kiley Sok] Add java 17 to changes
[Daniel Oliveira] [BEAM-13732] Switch x-lang BigQueryIO expansion service to GCP one.
[noreply] [BEAM-13858] Fix broken github action on :sdks:go:examples:wordCount
[Kiley Sok] add jira for runner v2
[noreply] [BEAM-13732] Go SDK BigQuery IO wrapper. Initial implementation.
[noreply] [BEAM-13732] Add example for Go BigQuery IO wrapper. (#16786)
[noreply] Update CHANGES.md with Go SDK milestones. (#16787)
[noreply] [BEAM-13193] Allow BeamFnDataOutboundObserver to flush elements.
------------------------------------------
[...truncated 4.28 MB...]
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.clear(LocalCache.java:4270)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalManualCache.invalidateAll(LocalCache.java:4909)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory.close(DefaultJobBundleFactory.java:319)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.fnexecution.control.DefaultExecutableStageContext.close(DefaultExecutableStageContext.java:43)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.fnexecution.control.ReferenceCountingExecutableStageContextFactory$WrappedContext.closeActual(ReferenceCountingExecutableStageContextFactory.java:212)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.fnexecution.control.ReferenceCountingExecutableStageContextFactory$WrappedContext.access$200(ReferenceCountingExecutableStageContextFactory.java:188)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.fnexecution.control.ReferenceCountingExecutableStageContextFactory.release(ReferenceCountingExecutableStageContextFactory.java:177)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.fnexecution.control.ReferenceCountingExecutableStageContextFactory.scheduleRelease(ReferenceCountingExecutableStageContextFactory.java:136)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.fnexecution.control.ReferenceCountingExecutableStageContextFactory.access$300(ReferenceCountingExecutableStageContextFactory.java:48)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.fnexecution.control.ReferenceCountingExecutableStageContextFactory$WrappedContext.close(ReferenceCountingExecutableStageContextFactory.java:208)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.spark.translation.SparkExecutableStageFunction.$closeResource(SparkExecutableStageFunction.java:210)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.spark.translation.SparkExecutableStageFunction.call(SparkExecutableStageFunction.java:210)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.spark.translation.SparkExecutableStageFunction.call(SparkExecutableStageFunction.java:81)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$4$1.apply(JavaRDDLike.scala:153)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$4$1.apply(JavaRDDLike.scala:153)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:310)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:310)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.scheduler.Task.run(Task.scala:123)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:411)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:417)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat java.lang.Thread.run(Thread.java:748)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/09 06:13:18 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job BeamApp-jenkins-0209061233-85cdcd21_ac71476a-1ecb-47c4-b83d-65516ff44695 finished.'
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:576 Job state changed to DONE
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.052184104919433594 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.048951148986816406 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.06571555137634277 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 12 files in 0.05951285362243652 seconds.
PASSED [ 95%]
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
-------------------------------- live log call ---------------------------------
INFO root:pipeline.py:188 Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_column_DataFrame_140653334113424], ComputedExpression[set_index_DataFrame_140652860429200], ComputedExpression[pre_combine_sum_DataFrame_140655465735568]]:140654686059792]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_140654686128400]}, partitioning=Arbitrary, ops=[ComputedExpression[set_column_DataFrame_140653334113424], ComputedExpression[set_index_DataFrame_140652860429200], ComputedExpression[pre_combine_sum_DataFrame_140655465735568]], outputs={PlaceholderExpression[placeholder_DataFrame_140654686128400], ComputedExpression[pre_combine_sum_DataFrame_140655465735568]}]
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[post_combine_sum_DataFrame_140654685895056]]:140654683215952]> for Stage[inputs={ComputedExpression[pre_combine_sum_DataFrame_140655465735568]}, partitioning=Index, ops=[ComputedExpression[post_combine_sum_DataFrame_140654685895056]], outputs={ComputedExpression[post_combine_sum_DataFrame_140654685895056]}]
INFO apache_beam.io.fileio:fileio.py:555 Added temporary directory /tmp/.tempdb01baa9-e5f4-4998-8210-00cb25cafc8e
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.7_sdk:2.37.0.dev
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function annotate_downstream_side_inputs at 0x7fecbd073710> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function fix_side_input_pcoll_coders at 0x7fecbd073830> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function pack_combiners at 0x7fecbd073d40> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function lift_combiners at 0x7fecbd073dd0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_sdf at 0x7fecbd073f80> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_gbk at 0x7fecbd06c050> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sink_flattens at 0x7fecbd06c170> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function greedily_fuse at 0x7fecbd06c200> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function read_to_impulse at 0x7fecbd06c290> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function impulse_to_input at 0x7fecbd06c320> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sort_stages at 0x7fecbd06c560> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function setup_timer_mapping at 0x7fecbd06c4d0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function populate_data_channel_coders at 0x7fecbd06c5f0> ====================
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 100
INFO apache_beam.runners.portability.fn_api_runner.worker_handlers:worker_handlers.py:894 Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7fec4bb7f890> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((ref_AppliedPTransform_Read-Read-Impulse_4)+(ref_AppliedPTransform_Read-Read-Map-lambda-at-iobase-py-898-_5))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((((ref_PCollection_PCollection_2_split/Read)+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Split_8))+(ref_AppliedPTransform_ToRows_9))+(ref_AppliedPTransform_BatchElements-words-BatchElements-ParDo-_GlobalWindowsBatchingDoFn-_12))+(ref_AppliedPTransform_BatchElements-words-Map-lambda-at-schemas-py-140-_13))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-set_column_DataFr_16))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-set_column_DataFr_18))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-post_combine_sum__21))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-post_combine_sum__22))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-post_combine_sum__23))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-post_combine_sum__26))+(ToPCollection(df) - /tmp/tmp5b56pa9y.result/[ComputedExpression[post_combine_sum_DataFrame_140654685895056]]:140654683215952/CoGroupByKey/CoGroupByKeyImpl/Flatten/Transcode/0))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-post_combine_sum__27))+(ToPCollection(df) - /tmp/tmp5b56pa9y.result/[ComputedExpression[post_combine_sum_DataFrame_140654685895056]]:140654683215952/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((ToPCollection(df) - /tmp/tmp5b56pa9y.result/[ComputedExpression[post_combine_sum_DataFrame_140654685895056]]:140654683215952/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-post_combine_sum__29))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-post_combine_sum__30))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-post_combine_sum__31))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp5b56pa9y-result-ComputedExpression-post_combine_sum__33))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp5b56pa9y-result-WriteToFiles-ParDo-_WriteUnshardedRe_37))+(ref_AppliedPTransform_Unbatch-post_combine_sum_DataFrame_140654685895056-with-indexes-ParDo-_Unbatch_46))+(WriteToPandas(df) - /tmp/tmp5b56pa9y.result/WriteToFiles/Flatten/Write/0))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp5b56pa9y-result-WriteToFiles-ParDo-_AppendShardedDes_38))+(WriteToPandas(df) - /tmp/tmp5b56pa9y.result/WriteToFiles/GroupRecordsByDestinationAndShard/Write))+(ref_AppliedPTransform_Filter-lambda-at-wordcount-py-80-_47))+(ref_AppliedPTransform_Map-lambda-at-wordcount-py-81-_48))+(ref_AppliedPTransform_Map-print-_49)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp5b56pa9y.result/WriteToFiles/GroupRecordsByDestinationAndShard/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp5b56pa9y-result-WriteToFiles-ParDo-_WriteShardedReco_40))+(WriteToPandas(df) - /tmp/tmp5b56pa9y.result/WriteToFiles/Flatten/Write/1)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp5b56pa9y.result/WriteToFiles/Flatten/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp5b56pa9y-result-WriteToFiles-Map-lambda-at-fileio-py_42))+(WriteToPandas(df) - /tmp/tmp5b56pa9y.result/WriteToFiles/GroupTempFilesByDestination/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running (WriteToPandas(df) - /tmp/tmp5b56pa9y.result/WriteToFiles/GroupTempFilesByDestination/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp5b56pa9y-result-WriteToFiles-ParDo-_MoveTempFilesInt_44)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file /tmp/.tempdb01baa9-e5f4-4998-8210-00cb25cafc8e/8786612676916981952_1e614b0d-cd48-4893-96e0-4f9b8172545f to dir: /tmp as tmp5b56pa9y.result-00000-of-00001. Res: FileResult(file_name='/tmp/.tempdb01baa9-e5f4-4998-8210-00cb25cafc8e/8786612676916981952_1e614b0d-cd48-4893-96e0-4f9b8172545f', shard_index=-1, total_shards=0, window=GlobalWindow, pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window GlobalWindow
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
PASSED [100%]
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2470: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:632: FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py37.xml> -
===== 22 passed, 1 skipped, 5181 deselected, 39 warnings in 489.10 seconds =====
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644386886.385307614","description":"Error received from peer ipv6:[::1]:42211","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644386905.323783805","description":"Error received from peer ipv6:[::1]:35633","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644387092.404645290","description":"Error received from peer ipv6:[::1]:46391","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 248
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py38:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 12m 55s
79 actionable tasks: 50 executed, 27 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/utokwlwz2hzp6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Spark #17
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/17/display/redirect?page=changes>
Changes:
[noreply] Update README.md
[marco.robles] Update README with latest PreCommit Jobs
[marco.robles] Update Postcommit jobs with latest jobs
[marco.robles] Update Performace job tests in readme
[marco.robles] update load job tests with latest updates
[marco.robles] update other jobs test with latest updates
[marco.robles] mismatch links fix
[marco.robles] update trigger phrase for some postCommit jobs
[marco.robles] correct trigger phrases in readme
[marco.robles] add pending jobs to readme
[noreply] Update README.md
[marco.robles] fix broken links in jobs & remove the invalid ones
[noreply] [BEAM-12914] Add missing 3.9 opcodes to type inference. (#16761)
[noreply] [BEAM-13321] Initial BigQueryIO externalization. (#16489)
[noreply] [BEAM-13193] Enable process bundle response elements embedding in Java
[noreply] [BEAM-13830] added a debeziumio_expansion_addr flag to GoSDK (#16780)
[noreply] Apply spotless. (#16783)
------------------------------------------
[...truncated 2.94 MB...]
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5884765099199580466_da979cac-ed61-4429-b4c7-42581d49f7a6 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5884765099199580466_da979cac-ed61-4429-b4c7-42581d49f7a6', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/7246766431021381326_0620a205-2cf2-4d95-9301-085bf2c89fdf to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/7246766431021381326_0620a205-2cf2-4d95-9301-085bf2c89fdf', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5294119421961123534_f0fe3cd9-0cf8-498e-adca-07c80a02a6c6 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5294119421961123534_f0fe3cd9-0cf8-498e-adca-07c80a02a6c6', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5884765099199580466_e4a149d6-9b3d-4eb1-a272-17c53f211294 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5884765099199580466_e4a149d6-9b3d-4eb1-a272-17c53f211294', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5294119421961123534_6f6a2051-2eda-4c00-80dc-e6f264952f5e to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5294119421961123534_6f6a2051-2eda-4c00-80dc-e6f264952f5e', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/7246766431021381326_cdfc101b-55e9-4816-98e9-9e5d94af4a15 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/7246766431021381326_cdfc101b-55e9-4816-98e9-9e5d94af4a15', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5884765099199580466_9764c53b-0cea-4a45-bbe8-c6fcf75024b4 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5884765099199580466_9764c53b-0cea-4a45-bbe8-c6fcf75024b4', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5294119421961123534_71b5b699-640d-4c8b-b1f3-1b5c83d0491b to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5294119421961123534_71b5b699-640d-4c8b-b1f3-1b5c83d0491b', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/7246766431021381326_0fe5715a-690c-4b6c-8138-2f2d87a1454d to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/7246766431021381326_0fe5715a-690c-4b6c-8138-2f2d87a1454d', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5884765099199580466_2bff6c79-dca2-4b75-828a-93904f5dfba0 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5884765099199580466_2bff6c79-dca2-4b75-828a-93904f5dfba0', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/7246766431021381326_37685f48-480a-4261-b08e-c31f619be3da to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/7246766431021381326_37685f48-480a-4261-b08e-c31f619be3da', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5294119421961123534_fc3c785e-9616-4d4b-9c18-ba82a2b5d3c5 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-eb5cbd3c-e294-4d8c-bc4a-33dfff4c7bcd as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.tempfe147917-a5f1-49df-b00d-9530daf11668/5294119421961123534_fc3c785e-9616-4d4b-9c18-ba82a2b5d3c5', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356220800.0, 1356307200.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356307200.0, 1356393600.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356393600.0, 1356480000.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.04474663734436035 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.05949974060058594 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.035530805587768555 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/09 00:19:31 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:external:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'payload: "\\n\\021\\n\\017localhost:39515"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bytes:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:string_utf8:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:kv:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:244 No more requests from control plane
ERROR apache_beam.runners.worker.data_plane:data_plane.py:641 Failed to read inputs in the data plane.
Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644365971.913068201","description":"Error received from peer ipv6:[::1]:35087","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bool:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:245 SDK Harness waiting for in-flight requests to complete
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:varint:v1"'
INFO apache_beam.runners.worker.data_plane:data_plane.py:782 Closing all cached grpc data channels.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:double:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:826 Closing all cached gRPC state handlers.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:iterable:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:257 Done consuming work.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:timer:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:interval_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:length_prefix:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:global_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:param_windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:state_backed_iterable:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:custom_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:row:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:sharded_key:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:progress_reporting:v0"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:harness_monitoring_infos:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:worker_status:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:combinefn:packed_python:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:version:sdk_base:apache/beam_python3.7_sdk:2.37.0.dev"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:to_string:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b''
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/09 00:19:31 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/09 00:19:31 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job BeamApp-jenkins-0209001909-c1dee451_5c68c5a1-adc4-4305-90f6-e265e5f799a3 finished.'
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:576 Job state changed to DONE
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.04782867431640625 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.06621885299682617 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.03997087478637695 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 12 files in 0.04198288917541504 seconds.
PASSED [ 95%]
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
-------------------------------- live log call ---------------------------------
INFO root:pipeline.py:188 Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_column_DataFrame_140089576386768], ComputedExpression[set_index_DataFrame_140089566225744], ComputedExpression[pre_combine_sum_DataFrame_140089565607248]]:140089150292496]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_140090512736464]}, partitioning=Arbitrary, ops=[ComputedExpression[set_column_DataFrame_140089576386768], ComputedExpression[set_index_DataFrame_140089566225744], ComputedExpression[pre_combine_sum_DataFrame_140089565607248]], outputs={ComputedExpression[pre_combine_sum_DataFrame_140089565607248], PlaceholderExpression[placeholder_DataFrame_140090512736464]}]
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[post_combine_sum_DataFrame_140089565606480]]:140089150293776]> for Stage[inputs={ComputedExpression[pre_combine_sum_DataFrame_140089565607248]}, partitioning=Index, ops=[ComputedExpression[post_combine_sum_DataFrame_140089565606480]], outputs={ComputedExpression[post_combine_sum_DataFrame_140089565606480]}]
INFO apache_beam.io.fileio:fileio.py:555 Added temporary directory /tmp/.temp728d4c55-178f-40ca-8476-af0ac8c41834
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.7_sdk:2.37.0.dev
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function annotate_downstream_side_inputs at 0x7f6929c38710> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function fix_side_input_pcoll_coders at 0x7f6929c38830> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function pack_combiners at 0x7f6929c38d40> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function lift_combiners at 0x7f6929c38dd0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_sdf at 0x7f6929c38f80> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_gbk at 0x7f6929c35050> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sink_flattens at 0x7f6929c35170> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function greedily_fuse at 0x7f6929c35200> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function read_to_impulse at 0x7f6929c35290> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function impulse_to_input at 0x7f6929c35320> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sort_stages at 0x7f6929c35560> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function setup_timer_mapping at 0x7f6929c354d0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function populate_data_channel_coders at 0x7f6929c355f0> ====================
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 100
INFO apache_beam.runners.portability.fn_api_runner.worker_handlers:worker_handlers.py:894 Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7f690c671c90> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((ref_AppliedPTransform_Read-Read-Impulse_4)+(ref_AppliedPTransform_Read-Read-Map-lambda-at-iobase-py-898-_5))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((((ref_PCollection_PCollection_2_split/Read)+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Split_8))+(ref_AppliedPTransform_ToRows_9))+(ref_AppliedPTransform_BatchElements-words-BatchElements-ParDo-_GlobalWindowsBatchingDoFn-_12))+(ref_AppliedPTransform_BatchElements-words-Map-lambda-at-schemas-py-140-_13))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-set_column_DataFr_16))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-set_column_DataFr_18))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-post_combine_sum__21))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-post_combine_sum__22))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-post_combine_sum__23))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-post_combine_sum__26))+(ToPCollection(df) - /tmp/tmp4vy16c11.result/[ComputedExpression[post_combine_sum_DataFrame_140089565606480]]:140089150293776/CoGroupByKey/CoGroupByKeyImpl/Flatten/Transcode/0))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-post_combine_sum__27))+(ToPCollection(df) - /tmp/tmp4vy16c11.result/[ComputedExpression[post_combine_sum_DataFrame_140089565606480]]:140089150293776/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((ToPCollection(df) - /tmp/tmp4vy16c11.result/[ComputedExpression[post_combine_sum_DataFrame_140089565606480]]:140089150293776/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-post_combine_sum__29))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-post_combine_sum__30))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-post_combine_sum__31))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp4vy16c11-result-ComputedExpression-post_combine_sum__33))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp4vy16c11-result-WriteToFiles-ParDo-_WriteUnshardedRe_37))+(ref_AppliedPTransform_Unbatch-post_combine_sum_DataFrame_140089565606480-with-indexes-ParDo-_Unbatch_46))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp4vy16c11-result-WriteToFiles-ParDo-_AppendShardedDes_38))+(WriteToPandas(df) - /tmp/tmp4vy16c11.result/WriteToFiles/Flatten/Write/0))+(WriteToPandas(df) - /tmp/tmp4vy16c11.result/WriteToFiles/GroupRecordsByDestinationAndShard/Write))+(ref_AppliedPTransform_Filter-lambda-at-wordcount-py-80-_47))+(ref_AppliedPTransform_Map-lambda-at-wordcount-py-81-_48))+(ref_AppliedPTransform_Map-print-_49)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp4vy16c11.result/WriteToFiles/GroupRecordsByDestinationAndShard/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp4vy16c11-result-WriteToFiles-ParDo-_WriteShardedReco_40))+(WriteToPandas(df) - /tmp/tmp4vy16c11.result/WriteToFiles/Flatten/Write/1)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp4vy16c11.result/WriteToFiles/Flatten/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp4vy16c11-result-WriteToFiles-Map-lambda-at-fileio-py_42))+(WriteToPandas(df) - /tmp/tmp4vy16c11.result/WriteToFiles/GroupTempFilesByDestination/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running (WriteToPandas(df) - /tmp/tmp4vy16c11.result/WriteToFiles/GroupTempFilesByDestination/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp4vy16c11-result-WriteToFiles-ParDo-_MoveTempFilesInt_44)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file /tmp/.temp728d4c55-178f-40ca-8476-af0ac8c41834/8470158733744207580_a40008a5-c880-49c9-98d8-04bc0f7b52ea to dir: /tmp as tmp4vy16c11.result-00000-of-00001. Res: FileResult(file_name='/tmp/.temp728d4c55-178f-40ca-8476-af0ac8c41834/8470158733744207580_a40008a5-c880-49c9-98d8-04bc0f7b52ea', shard_index=-1, total_shards=0, window=GlobalWindow, pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window GlobalWindow
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
PASSED [100%]
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2470: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:632: FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py37.xml> -
===== 22 passed, 1 skipped, 5181 deselected, 39 warnings in 234.14 seconds =====
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644365969.001600801","description":"Error received from peer ipv6:[::1]:37757","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 248
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py38:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 7m 20s
79 actionable tasks: 52 executed, 25 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/bwatiqyrhf3pa
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Spark #16
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/16/display/redirect?page=changes>
Changes:
[mmack] [BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk
[Pablo Estrada] Output successful rows from BQ Streaming Inserts
[schapman] BEAM-13439 Type annotation for ptransform_fn
[noreply] [BEAM-13606] Fail bundles with failed BigTable mutations (#16751)
------------------------------------------
[...truncated 2.87 MB...]
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 0
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:164 Creating insecure control channel for localhost:41753.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:172 Control channel established.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:215 Initializing SDKHarness with unbounded number of workers.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 131-1'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: getProcessBundleDescriptor request with id 131-2'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:807 Creating insecure state channel for localhost:35105.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:814 State channel established.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: getProcessBundleDescriptor request with id 131-3'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: getProcessBundleDescriptor request with id 131-4'
INFO apache_beam.runners.worker.data_plane:data_plane.py:750 Creating client data channel for localhost:38727
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:02 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.'
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5071369618668368078_a891fb01-ac41-4721-8d6e-0764d840174e to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5071369618668368078_a891fb01-ac41-4721-8d6e-0764d840174e', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/2384709940919684914_78177a81-2072-4022-b70b-8cbb7034ffcc to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/2384709940919684914_78177a81-2072-4022-b70b-8cbb7034ffcc', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5896756308365160654_01932417-e0a2-40f2-9139-227fac3a5b0f to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5896756308365160654_01932417-e0a2-40f2-9139-227fac3a5b0f', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/2384709940919684914_bbbfe2a6-7fd2-444e-ab9f-45dd84297c3d to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/2384709940919684914_bbbfe2a6-7fd2-444e-ab9f-45dd84297c3d', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5896756308365160654_c5453e23-5e99-459f-8a65-7f3ab3b8f8c1 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5896756308365160654_c5453e23-5e99-459f-8a65-7f3ab3b8f8c1', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5071369618668368078_b65aeea1-226e-4556-bf32-9975786661bc to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5071369618668368078_b65aeea1-226e-4556-bf32-9975786661bc', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5896756308365160654_b560b230-5392-440d-96d2-304130d3df9a to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5896756308365160654_b560b230-5392-440d-96d2-304130d3df9a', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/2384709940919684914_93a4795d-a61a-4d60-933f-0d2901c8db7d to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/2384709940919684914_93a4795d-a61a-4d60-933f-0d2901c8db7d', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5071369618668368078_cbdc0846-7ee0-4065-b252-7eecc9b2a478 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5071369618668368078_cbdc0846-7ee0-4065-b252-7eecc9b2a478', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5896756308365160654_a7576540-640b-43aa-85a1-388816a5ebc6 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5896756308365160654_a7576540-640b-43aa-85a1-388816a5ebc6', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5071369618668368078_cc6d5b64-697d-4b2d-b81b-2147e5547b04 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/5071369618668368078_cc6d5b64-697d-4b2d-b81b-2147e5547b04', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/2384709940919684914_b87081a1-09e1-4a4f-857c-63cf941b2b02 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-0ba0c961-7e10-44e5-85c1-59b9b2639b7b as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp0319c52e-3a09-43c2-9119-41a005cf7c13/2384709940919684914_b87081a1-09e1-4a4f-857c-63cf941b2b02', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356307200.0, 1356393600.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356220800.0, 1356307200.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356393600.0, 1356480000.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.04423856735229492 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.048372745513916016 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.04070782661437988 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:03 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:external:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'payload: "\\n\\021\\n\\017localhost:45383"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bytes:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:string_utf8:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:kv:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:244 No more requests from control plane
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bool:v1"'
ERROR apache_beam.runners.worker.data_plane:data_plane.py:641 Failed to read inputs in the data plane.
Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644343923.275773340","description":"Error received from peer ipv6:[::1]:38727","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:245 SDK Harness waiting for in-flight requests to complete
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:varint:v1"'
INFO apache_beam.runners.worker.data_plane:data_plane.py:782 Closing all cached grpc data channels.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:double:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:826 Closing all cached gRPC state handlers.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:iterable:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:timer:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:257 Done consuming work.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:interval_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:length_prefix:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:global_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:param_windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:state_backed_iterable:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:custom_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:row:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:sharded_key:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:progress_reporting:v0"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:harness_monitoring_infos:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:worker_status:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:combinefn:packed_python:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:version:sdk_base:apache/beam_python3.7_sdk:2.37.0.dev"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:to_string:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b''
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:03 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 18:12:03 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job BeamApp-jenkins-0208181137-3c2d3c4f_da091925-d871-4979-8f69-111f68f2a085 finished.'
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:576 Job state changed to DONE
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.0433964729309082 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.05379056930541992 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.029974937438964844 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 12 files in 0.03670692443847656 seconds.
PASSED [ 95%]
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
-------------------------------- live log call ---------------------------------
INFO root:pipeline.py:188 Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_column_DataFrame_139909278743248], ComputedExpression[set_index_DataFrame_139909278743760], ComputedExpression[pre_combine_sum_DataFrame_139909275999248]]:139908864531152]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_139908941668944]}, partitioning=Arbitrary, ops=[ComputedExpression[set_column_DataFrame_139909278743248], ComputedExpression[set_index_DataFrame_139909278743760], ComputedExpression[pre_combine_sum_DataFrame_139909275999248]], outputs={ComputedExpression[pre_combine_sum_DataFrame_139909275999248], PlaceholderExpression[placeholder_DataFrame_139908941668944]}]
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[post_combine_sum_DataFrame_139909275999568]]:139909277633424]> for Stage[inputs={ComputedExpression[pre_combine_sum_DataFrame_139909275999248]}, partitioning=Index, ops=[ComputedExpression[post_combine_sum_DataFrame_139909275999568]], outputs={ComputedExpression[post_combine_sum_DataFrame_139909275999568]}]
INFO apache_beam.io.fileio:fileio.py:555 Added temporary directory /tmp/.tempc0aee8a7-6649-446b-9317-70b170a29039
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.7_sdk:2.37.0.dev
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function annotate_downstream_side_inputs at 0x7f3f16c89710> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function fix_side_input_pcoll_coders at 0x7f3f16c89830> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function pack_combiners at 0x7f3f16c89d40> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function lift_combiners at 0x7f3f16c89dd0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_sdf at 0x7f3f16c89f80> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_gbk at 0x7f3f16c83050> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sink_flattens at 0x7f3f16c83170> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function greedily_fuse at 0x7f3f16c83200> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function read_to_impulse at 0x7f3f16c83290> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function impulse_to_input at 0x7f3f16c83320> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sort_stages at 0x7f3f16c83560> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function setup_timer_mapping at 0x7f3f16c834d0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function populate_data_channel_coders at 0x7f3f16c835f0> ====================
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 100
INFO apache_beam.runners.portability.fn_api_runner.worker_handlers:worker_handlers.py:894 Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7f3eb5995a50> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((ref_AppliedPTransform_Read-Read-Impulse_4)+(ref_AppliedPTransform_Read-Read-Map-lambda-at-iobase-py-898-_5))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((((ref_PCollection_PCollection_2_split/Read)+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Split_8))+(ref_AppliedPTransform_ToRows_9))+(ref_AppliedPTransform_BatchElements-words-BatchElements-ParDo-_GlobalWindowsBatchingDoFn-_12))+(ref_AppliedPTransform_BatchElements-words-Map-lambda-at-schemas-py-140-_13))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-set_column_DataFr_16))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-set_column_DataFr_18))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-post_combine_sum__21))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-post_combine_sum__22))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-post_combine_sum__23))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-post_combine_sum__26))+(ToPCollection(df) - /tmp/tmpvohk5v91.result/[ComputedExpression[post_combine_sum_DataFrame_139909275999568]]:139909277633424/CoGroupByKey/CoGroupByKeyImpl/Flatten/Transcode/0))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-post_combine_sum__27))+(ToPCollection(df) - /tmp/tmpvohk5v91.result/[ComputedExpression[post_combine_sum_DataFrame_139909275999568]]:139909277633424/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((ToPCollection(df) - /tmp/tmpvohk5v91.result/[ComputedExpression[post_combine_sum_DataFrame_139909275999568]]:139909277633424/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-post_combine_sum__29))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-post_combine_sum__30))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-post_combine_sum__31))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvohk5v91-result-ComputedExpression-post_combine_sum__33))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvohk5v91-result-WriteToFiles-ParDo-_WriteUnshardedRe_37))+(ref_AppliedPTransform_Unbatch-post_combine_sum_DataFrame_139909275999568-with-indexes-ParDo-_Unbatch_46))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvohk5v91-result-WriteToFiles-ParDo-_AppendShardedDes_38))+(WriteToPandas(df) - /tmp/tmpvohk5v91.result/WriteToFiles/Flatten/Write/0))+(WriteToPandas(df) - /tmp/tmpvohk5v91.result/WriteToFiles/GroupRecordsByDestinationAndShard/Write))+(ref_AppliedPTransform_Filter-lambda-at-wordcount-py-80-_47))+(ref_AppliedPTransform_Map-lambda-at-wordcount-py-81-_48))+(ref_AppliedPTransform_Map-print-_49)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmpvohk5v91.result/WriteToFiles/GroupRecordsByDestinationAndShard/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvohk5v91-result-WriteToFiles-ParDo-_WriteShardedReco_40))+(WriteToPandas(df) - /tmp/tmpvohk5v91.result/WriteToFiles/Flatten/Write/1)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmpvohk5v91.result/WriteToFiles/Flatten/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvohk5v91-result-WriteToFiles-Map-lambda-at-fileio-py_42))+(WriteToPandas(df) - /tmp/tmpvohk5v91.result/WriteToFiles/GroupTempFilesByDestination/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running (WriteToPandas(df) - /tmp/tmpvohk5v91.result/WriteToFiles/GroupTempFilesByDestination/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvohk5v91-result-WriteToFiles-ParDo-_MoveTempFilesInt_44)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file /tmp/.tempc0aee8a7-6649-446b-9317-70b170a29039/943171195407538307_ee49a162-a000-4ca1-9134-4c208b10e635 to dir: /tmp as tmpvohk5v91.result-00000-of-00001. Res: FileResult(file_name='/tmp/.tempc0aee8a7-6649-446b-9317-70b170a29039/943171195407538307_ee49a162-a000-4ca1-9134-4c208b10e635', shard_index=-1, total_shards=0, window=GlobalWindow, pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window GlobalWindow
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
PASSED [100%]
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2470: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:632: FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py37.xml> -
===== 22 passed, 1 skipped, 5178 deselected, 39 warnings in 225.23 seconds =====
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 248
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py38:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 5m 55s
79 actionable tasks: 51 executed, 26 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/ggcrlao7f4xzm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Spark #15
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/15/display/redirect?page=changes>
Changes:
[Ismaël MejÃa] [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1
[mmack] [BEAM-13840] Fix usage of legacy rawtypes in AWS modules
------------------------------------------
[...truncated 2.97 MB...]
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:combinefn:packed_python:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:version:sdk_base:apache/beam_python3.7_sdk:2.37.0.dev"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:to_string:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b''
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:244 No more requests from control plane
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:245 SDK Harness waiting for in-flight requests to complete
INFO apache_beam.runners.worker.data_plane:data_plane.py:782 Closing all cached grpc data channels.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:826 Closing all cached gRPC state handlers.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 12:18:52 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 12:18:52 ERROR org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.SerializingExecutor: Exception while executing runnable org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@13c8b75c'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'java.lang.IllegalStateException: call already closed'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.com.google.common.base.Preconditions.checkState(Preconditions.java:510)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ServerCallImpl.closeInternal(ServerCallImpl.java:214)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ServerCallImpl.close(ServerCallImpl.java:207)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:395)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.runners.fnexecution.state.GrpcStateService$Inbound.onCompleted(GrpcStateService.java:150)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onHalfClose(ServerCalls.java:273)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:340)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:866)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat org.apache.beam.vendor.grpc.v1p43p2.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'\tat java.lang.Thread.run(Thread.java:748)'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:257 Done consuming work.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 12:18:52 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job BeamApp-jenkins-0208121800-4a6f1b45_442cfafe-55b8-450e-aa43-aeb05f4ac4fa finished.'
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:576 Job state changed to DONE
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.04415297508239746 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.0861210823059082 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.06580138206481934 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 12 files in 0.06737899780273438 seconds.
PASSED [ 95%]
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
-------------------------------- live log call ---------------------------------
INFO root:pipeline.py:188 Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_column_DataFrame_140122792144016], ComputedExpression[set_index_DataFrame_140123987523024], ComputedExpression[pre_combine_sum_DataFrame_140122778649424]]:140123802482448]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_140122793011984]}, partitioning=Arbitrary, ops=[ComputedExpression[set_column_DataFrame_140122792144016], ComputedExpression[set_index_DataFrame_140123987523024], ComputedExpression[pre_combine_sum_DataFrame_140122778649424]], outputs={PlaceholderExpression[placeholder_DataFrame_140122793011984], ComputedExpression[pre_combine_sum_DataFrame_140122778649424]}]
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[post_combine_sum_DataFrame_140122778650256]]:140123802481744]> for Stage[inputs={ComputedExpression[pre_combine_sum_DataFrame_140122778649424]}, partitioning=Index, ops=[ComputedExpression[post_combine_sum_DataFrame_140122778650256]], outputs={ComputedExpression[post_combine_sum_DataFrame_140122778650256]}]
INFO apache_beam.io.fileio:fileio.py:555 Added temporary directory /tmp/.tempb258d2cb-fc14-4b01-b8dc-95e8aaa1b7c9
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.7_sdk:2.37.0.dev
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function annotate_downstream_side_inputs at 0x7f70f4fc2710> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function fix_side_input_pcoll_coders at 0x7f70f4fc2830> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function pack_combiners at 0x7f70f4fc2d40> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function lift_combiners at 0x7f70f4fc2dd0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_sdf at 0x7f70f4fc2f80> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_gbk at 0x7f70f4fbe050> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sink_flattens at 0x7f70f4fbe170> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function greedily_fuse at 0x7f70f4fbe200> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function read_to_impulse at 0x7f70f4fbe290> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function impulse_to_input at 0x7f70f4fbe320> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sort_stages at 0x7f70f4fbe560> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function setup_timer_mapping at 0x7f70f4fbe4d0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function populate_data_channel_coders at 0x7f70f4fbe5f0> ====================
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 100
INFO apache_beam.runners.portability.fn_api_runner.worker_handlers:worker_handlers.py:894 Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7f70e179c410> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((ref_AppliedPTransform_Read-Read-Impulse_4)+(ref_AppliedPTransform_Read-Read-Map-lambda-at-iobase-py-898-_5))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((((ref_PCollection_PCollection_2_split/Read)+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Split_8))+(ref_AppliedPTransform_ToRows_9))+(ref_AppliedPTransform_BatchElements-words-BatchElements-ParDo-_GlobalWindowsBatchingDoFn-_12))+(ref_AppliedPTransform_BatchElements-words-Map-lambda-at-schemas-py-140-_13))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-set_column_DataFr_16))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-set_column_DataFr_18))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-post_combine_sum__21))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-post_combine_sum__22))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-post_combine_sum__23))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-post_combine_sum__26))+(ToPCollection(df) - /tmp/tmp7ylvm_4g.result/[ComputedExpression[post_combine_sum_DataFrame_140122778650256]]:140123802481744/CoGroupByKey/CoGroupByKeyImpl/Flatten/Transcode/0))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-post_combine_sum__27))+(ToPCollection(df) - /tmp/tmp7ylvm_4g.result/[ComputedExpression[post_combine_sum_DataFrame_140122778650256]]:140123802481744/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((ToPCollection(df) - /tmp/tmp7ylvm_4g.result/[ComputedExpression[post_combine_sum_DataFrame_140122778650256]]:140123802481744/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-post_combine_sum__29))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-post_combine_sum__30))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-post_combine_sum__31))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmp7ylvm_4g-result-ComputedExpression-post_combine_sum__33))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp7ylvm_4g-result-WriteToFiles-ParDo-_WriteUnshardedRe_37))+(ref_AppliedPTransform_Unbatch-post_combine_sum_DataFrame_140122778650256-with-indexes-ParDo-_Unbatch_46))+(WriteToPandas(df) - /tmp/tmp7ylvm_4g.result/WriteToFiles/Flatten/Write/0))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp7ylvm_4g-result-WriteToFiles-ParDo-_AppendShardedDes_38))+(WriteToPandas(df) - /tmp/tmp7ylvm_4g.result/WriteToFiles/GroupRecordsByDestinationAndShard/Write))+(ref_AppliedPTransform_Filter-lambda-at-wordcount-py-80-_47))+(ref_AppliedPTransform_Map-lambda-at-wordcount-py-81-_48))+(ref_AppliedPTransform_Map-print-_49)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp7ylvm_4g.result/WriteToFiles/GroupRecordsByDestinationAndShard/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp7ylvm_4g-result-WriteToFiles-ParDo-_WriteShardedReco_40))+(WriteToPandas(df) - /tmp/tmp7ylvm_4g.result/WriteToFiles/Flatten/Write/1)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmp7ylvm_4g.result/WriteToFiles/Flatten/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp7ylvm_4g-result-WriteToFiles-Map-lambda-at-fileio-py_42))+(WriteToPandas(df) - /tmp/tmp7ylvm_4g.result/WriteToFiles/GroupTempFilesByDestination/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running (WriteToPandas(df) - /tmp/tmp7ylvm_4g.result/WriteToFiles/GroupTempFilesByDestination/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmp7ylvm_4g-result-WriteToFiles-ParDo-_MoveTempFilesInt_44)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file /tmp/.tempb258d2cb-fc14-4b01-b8dc-95e8aaa1b7c9/3619949647721907564_36665319-fe2a-4a4c-8488-72b4fb5a3484 to dir: /tmp as tmp7ylvm_4g.result-00000-of-00001. Res: FileResult(file_name='/tmp/.tempb258d2cb-fc14-4b01-b8dc-95e8aaa1b7c9/3619949647721907564_36665319-fe2a-4a4c-8488-72b4fb5a3484', shard_index=-1, total_shards=0, window=GlobalWindow, pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window GlobalWindow
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
PASSED [100%]
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2470: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:632: FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py37.xml> -
===== 22 passed, 1 skipped, 5178 deselected, 39 warnings in 261.40 seconds =====
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644322653.151793105","description":"Error received from peer ipv6:[::1]:43939","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644322653.300619887","description":"Error received from peer ipv6:[::1]:33779","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 651, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644322664.024619350","description":"Error received from peer ipv6:[::1]:36681","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 248
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py38:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 13m 20s
79 actionable tasks: 55 executed, 22 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/nimuu7fazt3xe
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_Examples_Spark #14
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/14/display/redirect?page=changes>
Changes:
[noreply] [BEAM-13848] Update numpy intersphinx link (#16767)
[noreply] [release-23.6.0] Fix JIRA link for 2.36 blog (#16771)
[noreply] [BEAM-13647] Use role for Go worker binary. (#16729)
------------------------------------------
[...truncated 2.74 MB...]
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 WARN software.amazon.awssdk.regions.internal.util.EC2MetadataUtils: Unable to retrieve the requested metadata.'
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 0
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:164 Creating insecure control channel for localhost:45441.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:172 Control channel established.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:215 Initializing SDKHarness with unbounded number of workers.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 131-1'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: getProcessBundleDescriptor request with id 131-2'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:807 Creating insecure state channel for localhost:33677.
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:814 State channel established.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: getProcessBundleDescriptor request with id 131-3'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: getProcessBundleDescriptor request with id 131-4'
INFO apache_beam.runners.worker.data_plane:data_plane.py:750 Creating client data channel for localhost:46653
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:12 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.'
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/3458111738400001586_441bffc2-e5ff-4e63-befe-60945db02eac to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/3458111738400001586_441bffc2-e5ff-4e63-befe-60945db02eac', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/1631671061563887154_2f090387-2b28-4900-8680-8fc55c3b32ec to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/1631671061563887154_2f090387-2b28-4900-8680-8fc55c3b32ec', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/2296956298787467726_9898fa08-f2c2-45b2-9d2f-eabacbef6652 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00000-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/2296956298787467726_9898fa08-f2c2-45b2-9d2f-eabacbef6652', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/3458111738400001586_12609429-ca41-4cf3-aef7-7a6f3f3f5312 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/3458111738400001586_12609429-ca41-4cf3-aef7-7a6f3f3f5312', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/2296956298787467726_3df734cb-c2ea-48ef-bc12-d6969471c1db to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/2296956298787467726_3df734cb-c2ea-48ef-bc12-d6969471c1db', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/1631671061563887154_965ee60c-98c2-4b1a-b3dc-0e574fee4c34 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00001-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/1631671061563887154_965ee60c-98c2-4b1a-b3dc-0e574fee4c34', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/3458111738400001586_0fbcf48b-a361-4beb-8d08-bc3166f0e2a2 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/3458111738400001586_0fbcf48b-a361-4beb-8d08-bc3166f0e2a2', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/2296956298787467726_78999dcf-3040-4d1c-9f21-23077681f005 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/2296956298787467726_78999dcf-3040-4d1c-9f21-23077681f005', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/1631671061563887154_8b46a52d-be77-4e6a-8f5a-3fe1d8e555a7 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00002-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/1631671061563887154_8b46a52d-be77-4e6a-8f5a-3fe1d8e555a7', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/2296956298787467726_e777c774-1612-4699-9cea-7c316d897f65 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-23T00:00:00-2012-12-24T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/2296956298787467726_e777c774-1612-4699-9cea-7c316d897f65', shard_index=-1, total_shards=0, window=[1356220800.0, 1356307200.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/3458111738400001586_a56cbb27-172b-43eb-9cf3-21ce96377e26 to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-24T00:00:00-2012-12-25T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/3458111738400001586_a56cbb27-172b-43eb-9cf3-21ce96377e26', shard_index=-1, total_shards=0, window=[1356307200.0, 1356393600.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/1631671061563887154_68bcee19-5f58-484c-a1f6-ad1d003f39dd to dir: gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-90bed2d4-1817-43b0-9b32-27f42081ef17 as output.csv-2012-12-25T00:00:00-2012-12-26T00:00:00-00003-of-00004. Res: FileResult(file_name='gs://temp-storage-for-end-to-end-tests/temp-it/.temp2fd192ce-a2d3-44ea-91b1-b3d0fa9724c7/1631671061563887154_68bcee19-5f58-484c-a1f6-ad1d003f39dd', shard_index=-1, total_shards=0, window=[1356393600.0, 1356480000.0), pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356220800.0, 1356307200.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.03402066230773926 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356307200.0, 1356393600.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window [1356393600.0, 1356480000.0)
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.02733755111694336 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 0 files in 0.04362916946411133 seconds.
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:13 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:external:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'payload: "\\n\\021\\n\\017localhost:38827"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bytes:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:string_utf8:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:244 No more requests from control plane
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:kv:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:245 SDK Harness waiting for in-flight requests to complete
ERROR apache_beam.runners.worker.data_plane:data_plane.py:641 Failed to read inputs in the data plane.
Traceback (most recent call last):
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py",> line 634, in _read_inputs
for elements in elements_iterator:
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 426, in __next__
return self._next()
File "<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",> line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.CANCELLED
details = "Multiplexer hanging up"
debug_error_string = "{"created":"@1644300553.846580955","description":"Error received from peer ipv6:[::1]:46653","file":"src/core/lib/surface/call.cc","file_line":1074,"grpc_message":"Multiplexer hanging up","grpc_status":1}"
>
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:bool:v1"'
INFO apache_beam.runners.worker.data_plane:data_plane.py:782 Closing all cached grpc data channels.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:varint:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:826 Closing all cached gRPC state handlers.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:double:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:iterable:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:timer:v1"'
INFO apache_beam.runners.worker.sdk_worker:sdk_worker.py:257 Done consuming work.
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:interval_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:length_prefix:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:global_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:param_windowed_value:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:state_backed_iterable:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:custom_window:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:row:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:coder:sharded_key:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:progress_reporting:v0"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:harness_monitoring_infos:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:protocol:worker_status:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:combinefn:packed_python:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:version:sdk_base:apache/beam_python3.7_sdk:2.37.0.dev"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'capabilities: "beam:transform:to_string:v1"'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b''
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:13 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.'
INFO apache_beam.utils.subprocess_server:subprocess_server.py:125 b'22/02/08 06:09:13 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job BeamApp-jenkins-0208060851-7c24d6c1_d4747440-e045-4825-b17c-da879bae3a20 finished.'
INFO apache_beam.runners.portability.portable_runner:portable_runner.py:576 Job state changed to DONE
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.03216838836669922 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.050238609313964844 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 4 files in 0.03373980522155762 seconds.
INFO apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input
INFO apache_beam.io.gcp.gcsio:gcsio.py:575 Finished listing 12 files in 0.0411527156829834 seconds.
PASSED [ 95%]
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
-------------------------------- live log call ---------------------------------
INFO root:pipeline.py:188 Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[set_column_DataFrame_140677047418768], ComputedExpression[set_index_DataFrame_140677047678544], ComputedExpression[pre_combine_sum_DataFrame_140676091688464]]:140677047417680]> for Stage[inputs={PlaceholderExpression[placeholder_DataFrame_140677451204688]}, partitioning=Arbitrary, ops=[ComputedExpression[set_column_DataFrame_140677047418768], ComputedExpression[set_index_DataFrame_140677047678544], ComputedExpression[pre_combine_sum_DataFrame_140676091688464]], outputs={ComputedExpression[pre_combine_sum_DataFrame_140676091688464], PlaceholderExpression[placeholder_DataFrame_140677451204688]}]
INFO root:transforms.py:182 Computing dataframe stage <ComputeStage(PTransform) label=[[ComputedExpression[post_combine_sum_DataFrame_140676091687760]]:140677047418192]> for Stage[inputs={ComputedExpression[pre_combine_sum_DataFrame_140676091688464]}, partitioning=Index, ops=[ComputedExpression[post_combine_sum_DataFrame_140676091687760]], outputs={ComputedExpression[post_combine_sum_DataFrame_140676091687760]}]
INFO apache_beam.io.fileio:fileio.py:555 Added temporary directory /tmp/.tempaa80dd09-908e-4c53-b107-12cc145a4678
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.7_sdk:2.37.0.dev
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function annotate_downstream_side_inputs at 0x7ff1f1e74710> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function fix_side_input_pcoll_coders at 0x7ff1f1e74830> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function pack_combiners at 0x7ff1f1e74d40> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function lift_combiners at 0x7ff1f1e74dd0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_sdf at 0x7ff1f1e74f80> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function expand_gbk at 0x7ff1f1e76050> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sink_flattens at 0x7ff1f1e76170> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function greedily_fuse at 0x7ff1f1e76200> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function read_to_impulse at 0x7ff1f1e76290> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function impulse_to_input at 0x7ff1f1e76320> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function sort_stages at 0x7ff1f1e76560> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function setup_timer_mapping at 0x7ff1f1e764d0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:678 ==================== <function populate_data_channel_coders at 0x7ff1f1e765f0> ====================
INFO apache_beam.runners.worker.statecache:statecache.py:172 Creating state cache with size 100
INFO apache_beam.runners.portability.fn_api_runner.worker_handlers:worker_handlers.py:894 Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7ff1d4439310> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((ref_AppliedPTransform_Read-Read-Impulse_4)+(ref_AppliedPTransform_Read-Read-Map-lambda-at-iobase-py-898-_5))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction))+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/SplitAndSizeRestriction))+(ref_PCollection_PCollection_2_split/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((((ref_PCollection_PCollection_2_split/Read)+(Read/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/Process))+(ref_AppliedPTransform_Split_8))+(ref_AppliedPTransform_ToRows_9))+(ref_AppliedPTransform_BatchElements-words-BatchElements-ParDo-_GlobalWindowsBatchingDoFn-_12))+(ref_AppliedPTransform_BatchElements-words-Map-lambda-at-schemas-py-140-_13))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-set_column_DataFr_16))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-set_column_DataFr_18))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-post_combine_sum__21))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-post_combine_sum__22))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-post_combine_sum__23))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-post_combine_sum__26))+(ToPCollection(df) - /tmp/tmpvchc_jfq.result/[ComputedExpression[post_combine_sum_DataFrame_140676091687760]]:140677047418192/CoGroupByKey/CoGroupByKeyImpl/Flatten/Transcode/0))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-post_combine_sum__27))+(ToPCollection(df) - /tmp/tmpvchc_jfq.result/[ComputedExpression[post_combine_sum_DataFrame_140676091687760]]:140677047418192/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((((((((((((ToPCollection(df) - /tmp/tmpvchc_jfq.result/[ComputedExpression[post_combine_sum_DataFrame_140676091687760]]:140677047418192/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read)+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-post_combine_sum__29))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-post_combine_sum__30))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-post_combine_sum__31))+(ref_AppliedPTransform_ToPCollection-df---tmp-tmpvchc_jfq-result-ComputedExpression-post_combine_sum__33))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvchc_jfq-result-WriteToFiles-ParDo-_WriteUnshardedRe_37))+(ref_AppliedPTransform_Unbatch-post_combine_sum_DataFrame_140676091687760-with-indexes-ParDo-_Unbatch_46))+(WriteToPandas(df) - /tmp/tmpvchc_jfq.result/WriteToFiles/Flatten/Write/0))+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvchc_jfq-result-WriteToFiles-ParDo-_AppendShardedDes_38))+(WriteToPandas(df) - /tmp/tmpvchc_jfq.result/WriteToFiles/GroupRecordsByDestinationAndShard/Write))+(ref_AppliedPTransform_Filter-lambda-at-wordcount-py-80-_47))+(ref_AppliedPTransform_Map-lambda-at-wordcount-py-81-_48))+(ref_AppliedPTransform_Map-print-_49)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmpvchc_jfq.result/WriteToFiles/GroupRecordsByDestinationAndShard/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvchc_jfq-result-WriteToFiles-ParDo-_WriteShardedReco_40))+(WriteToPandas(df) - /tmp/tmpvchc_jfq.result/WriteToFiles/Flatten/Write/1)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running ((WriteToPandas(df) - /tmp/tmpvchc_jfq.result/WriteToFiles/Flatten/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvchc_jfq-result-WriteToFiles-Map-lambda-at-fileio-py_42))+(WriteToPandas(df) - /tmp/tmpvchc_jfq.result/WriteToFiles/GroupTempFilesByDestination/Write)
INFO apache_beam.runners.portability.fn_api_runner.fn_runner:fn_runner.py:621 Running (WriteToPandas(df) - /tmp/tmpvchc_jfq.result/WriteToFiles/GroupTempFilesByDestination/Read)+(ref_AppliedPTransform_WriteToPandas-df---tmp-tmpvchc_jfq-result-WriteToFiles-ParDo-_MoveTempFilesInt_44)
INFO apache_beam.io.fileio:fileio.py:642 Moving temporary file /tmp/.tempaa80dd09-908e-4c53-b107-12cc145a4678/3770419459992350901_cb01a105-a12c-4665-84fd-ff40e3af17a4 to dir: /tmp as tmpvchc_jfq.result-00000-of-00001. Res: FileResult(file_name='/tmp/.tempaa80dd09-908e-4c53-b107-12cc145a4678/3770419459992350901_cb01a105-a12c-4665-84fd-ff40e3af17a4', shard_index=-1, total_shards=0, window=GlobalWindow, pane=None, destination=None)
INFO apache_beam.io.fileio:fileio.py:665 Checking orphaned temporary files for destination None and window GlobalWindow
INFO apache_beam.io.fileio:fileio.py:678 Some files may be left orphaned in the temporary folder: []
PASSED [100%]
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
dataset_ref = client.dataset(unique_dataset_name, project=project)
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
apache_beam/examples/complete/game/hourly_team_score_it_test.py::HourlyTeamScoreIT::test_hourly_team_score_output_checksum_on_small_input
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME')
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_side_input_it_test.py::BigQuerySideInputIT::test_bigquery_side_input_it
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2470: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
| _PassThroughThenCleanup(files_to_remove_pcoll))
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
self.table_reference.projectId = pcoll.pipeline.options.view_as(
apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it
apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead.
table_ref = client.dataset(dataset_id).table(table_id)
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
return airline_df[at_top_airports].mean()
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/dataframe/io.py>:632: FutureWarning: WriteToFiles is experimental.
sink=lambda _: _WriteToPandasFileSink(
apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays
apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics
<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
p.options.view_as(GoogleCloudOptions).temp_location or
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/pytest_postCommitExamples-spark-py37.xml> -
===== 22 passed, 1 skipped, 5178 deselected, 39 warnings in 306.14 seconds =====
FAILURE: Build failed with an exception.
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 248
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py38:sparkExamples'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 7m 19s
79 actionable tasks: 50 executed, 27 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/mn645mge6nx7w
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org