You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2019/12/09 12:53:20 UTC
Build failed in Jenkins: beam_PostCommit_Python_VR_Spark #1731
See <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/1731/display/redirect?page=changes>
Changes:
[michal.walenia] [BEAM-8895] Add BigQuery table name sanitization to BigQueryIOIT
[michal.walenia] [BEAM-8918] Split batch BQIOIT into avro and json using tests
------------------------------------------
[...truncated 1.55 MB...]
19/12/09 12:53:14 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:36339
19/12/09 12:53:14 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 12:53:14 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 12:53:14 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575895992.03_c97f5a38-9831-429e-90aa-52ea4b57eb68', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 12:53:14 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575895992.03', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:35417', 'job_port': u'0'}
19/12/09 12:53:14 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 12:53:14 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:38649.
19/12/09 12:53:14 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 259-1
19/12/09 12:53:14 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 12:53:14 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 12:53:14 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:46525.
19/12/09 12:53:14 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 12:53:14 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:46527
19/12/09 12:53:14 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 12:53:14 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 12:53:14 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 12:53:14 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 12:53:14 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 12:53:14 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 12:53:14 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 12:53:14 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 12:53:14 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 12:53:14 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 12:53:14 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 12:53:14 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 12:53:15 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 12:53:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 12:53:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:40829
19/12/09 12:53:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 12:53:15 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 12:53:15 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575895992.03_c97f5a38-9831-429e-90aa-52ea4b57eb68', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 12:53:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575895992.03', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:35417', 'job_port': u'0'}
19/12/09 12:53:15 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 12:53:15 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:37033.
19/12/09 12:53:15 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 260-1
19/12/09 12:53:15 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 12:53:15 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 12:53:15 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:34659.
19/12/09 12:53:15 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 12:53:15 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:45653
19/12/09 12:53:15 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 12:53:15 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 12:53:15 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 12:53:15 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 12:53:15 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 12:53:15 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 12:53:15 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 12:53:15 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 12:53:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 12:53:15 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 12:53:15 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 12:53:15 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 12:53:16 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 12:53:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 12:53:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:46305
19/12/09 12:53:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 12:53:16 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 12:53:16 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575895992.03_c97f5a38-9831-429e-90aa-52ea4b57eb68', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 12:53:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575895992.03', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:35417', 'job_port': u'0'}
19/12/09 12:53:16 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 12:53:16 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:33491.
19/12/09 12:53:16 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 12:53:16 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 12:53:16 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 261-1
19/12/09 12:53:16 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:39537.
19/12/09 12:53:16 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 12:53:16 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:46153
19/12/09 12:53:16 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 12:53:16 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 12:53:16 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 12:53:16 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 12:53:16 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 12:53:16 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 12:53:16 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 12:53:16 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 12:53:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 12:53:16 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 12:53:16 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 12:53:16 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 12:53:17 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 12:53:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 12:53:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:41155
19/12/09 12:53:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 12:53:17 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 12:53:17 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575895992.03_c97f5a38-9831-429e-90aa-52ea4b57eb68', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 12:53:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575895992.03', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:35417', 'job_port': u'0'}
19/12/09 12:53:17 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 12:53:17 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:37345.
19/12/09 12:53:17 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 12:53:17 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 12:53:17 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 262-1
19/12/09 12:53:17 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:38467.
19/12/09 12:53:17 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 12:53:17 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:46013
19/12/09 12:53:17 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 12:53:17 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 12:53:17 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 12:53:17 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 12:53:17 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 12:53:17 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 12:53:17 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 12:53:17 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 12:53:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 12:53:17 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 12:53:17 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 12:53:17 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job test_windowing_1575895992.03_c97f5a38-9831-429e-90aa-52ea4b57eb68 finished.
19/12/09 12:53:17 WARN org.apache.beam.runners.spark.SparkPipelineResult$BatchMode: Collecting monitoring infos is not implemented yet in Spark portable runner.
19/12/09 12:53:17 WARN org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService: Failed to remove job staging directory for token {"sessionId":"job_19057db3-6693-4158-bb4c-e570b713686b","basePath":"/tmp/sparktestI3AjBJ"}: {}
java.io.FileNotFoundException: /tmp/sparktestI3AjBJ/job_19057db3-6693-4158-bb4c-e570b713686b/MANIFEST (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:118)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:82)
at org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:252)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:88)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService.removeArtifacts(BeamFileSystemArtifactStagingService.java:92)
at org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver.lambda$createJobService$0(JobServerDriver.java:63)
at org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.lambda$run$0(InMemoryJobService.java:201)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.setState(JobInvocation.java:241)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.access$200(JobInvocation.java:48)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:110)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:96)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1058)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE
.
======================================================================
ERROR: test_pardo_state_with_custom_key_coder (__main__.SparkRunnerTest)
Tests that state requests work correctly when the key coder is an
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/portable_runner_test.py", line 231, in test_pardo_state_with_custom_key_coder
equal_to(expected))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
==================== Timed out after 60 seconds. ====================
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
# Thread: <Thread(wait_until_finish_read, started daemon 140595502864128)>
======================================================================
ERROR: test_pardo_timers (__main__.SparkRunnerTest)
# Thread: <Thread(Thread-119, started daemon 140595494471424)>
----------------------------------------------------------------------
Traceback (most recent call last):
# Thread: <_MainThread(MainThread, started 140596290995968)>
==================== Timed out after 60 seconds. ====================
File "apache_beam/runners/portability/fn_api_runner_test.py", line 330, in test_pardo_timers
assert_that(actual, equal_to(expected))
# Thread: <Thread(wait_until_finish_read, started daemon 140595007579904)>
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
# Thread: <Thread(Thread-125, started daemon 140595015972608)>
# Thread: <Thread(Thread-119, started daemon 140595494471424)>
# Thread: <Thread(wait_until_finish_read, started daemon 140595502864128)>
File "apache_beam/runners/portability/portable_ru# Thread: <_MainThread(MainThread, started 140596290995968)>
nner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
======================================================================
ERROR: test_sdf_with_watermark_tracking (__main__.SparkRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/fn_api_runner_test.py", line 502, in test_sdf_with_watermark_tracking
assert_that(actual, equal_to(list(''.join(data))))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 438, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline test_sdf_with_watermark_tracking_1575895983.48_ea2a82bc-014e-4c72-9363-80a6c5c6ce41 failed in state FAILED: java.lang.UnsupportedOperationException: The ActiveBundle does not have a registered bundle checkpoint handler.
----------------------------------------------------------------------
Ran 38 tests in 312.712s
FAILED (errors=3, skipped=9)
> Task :sdks:python:test-suites:portable:py2:sparkValidatesRunner FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build.gradle'> line: 196
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py2:sparkValidatesRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 8m 14s
60 actionable tasks: 47 executed, 13 from cache
Publishing build scan...
https://scans.gradle.com/s/kszdjin5vdoww
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_Python_VR_Spark
#1737
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/1737/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_VR_Spark #1736
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/1736/display/redirect?page=changes>
Changes:
[kcweaver] Version Flink job server container images
[kcweaver] [BEAM-8337] publish Flink job server container images
[ningk] [BEAM-7926] Data-centric Interactive Part1
[kcweaver] Get Flink version numbers from subdirectories
[kcweaver] Warn if Flink versions can't be listed.
------------------------------------------
[...truncated 1.55 MB...]
19/12/10 01:08:02 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:46883
19/12/10 01:08:02 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/10 01:08:02 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/10 01:08:02 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575940079.97_99e63056-ab2b-43ae-97e2-606351b62399', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/10 01:08:02 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575940079.97', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:58733', 'job_port': u'0'}
19/12/10 01:08:02 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/10 01:08:02 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:40145.
19/12/10 01:08:02 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/10 01:08:02 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/10 01:08:02 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 259-1
19/12/10 01:08:02 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:40853.
19/12/10 01:08:02 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/10 01:08:02 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:42997
19/12/10 01:08:02 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/10 01:08:02 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/10 01:08:02 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/10 01:08:02 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/10 01:08:02 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/10 01:08:02 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/10 01:08:02 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/10 01:08:02 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/10 01:08:02 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/10 01:08:02 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/10 01:08:02 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/10 01:08:03 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/10 01:08:03 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/10 01:08:03 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/10 01:08:03 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:41917
19/12/10 01:08:03 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/10 01:08:03 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/10 01:08:03 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575940079.97_99e63056-ab2b-43ae-97e2-606351b62399', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/10 01:08:03 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575940079.97', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:58733', 'job_port': u'0'}
19/12/10 01:08:03 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/10 01:08:03 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:39631.
19/12/10 01:08:03 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/10 01:08:03 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 260-1
19/12/10 01:08:03 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/10 01:08:03 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:40747.
19/12/10 01:08:03 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/10 01:08:03 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:32889
19/12/10 01:08:03 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/10 01:08:03 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/10 01:08:03 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/10 01:08:03 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/10 01:08:03 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/10 01:08:03 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/10 01:08:03 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/10 01:08:03 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/10 01:08:03 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/10 01:08:03 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/10 01:08:04 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/10 01:08:04 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/10 01:08:04 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/10 01:08:04 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/10 01:08:04 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:36863
19/12/10 01:08:04 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/10 01:08:04 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/10 01:08:04 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575940079.97_99e63056-ab2b-43ae-97e2-606351b62399', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/10 01:08:04 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575940079.97', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:58733', 'job_port': u'0'}
19/12/10 01:08:04 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/10 01:08:04 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:34651.
19/12/10 01:08:04 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/10 01:08:04 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 261-1
19/12/10 01:08:04 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/10 01:08:04 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:44799.
19/12/10 01:08:04 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/10 01:08:04 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:34641
19/12/10 01:08:04 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/10 01:08:04 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/10 01:08:04 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/10 01:08:04 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/10 01:08:04 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/10 01:08:04 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/10 01:08:04 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/10 01:08:04 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/10 01:08:04 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/10 01:08:04 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/10 01:08:04 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/10 01:08:04 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/10 01:08:05 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/10 01:08:05 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/10 01:08:05 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:38251
19/12/10 01:08:05 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/10 01:08:05 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/10 01:08:05 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575940079.97_99e63056-ab2b-43ae-97e2-606351b62399', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/10 01:08:05 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575940079.97', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:58733', 'job_port': u'0'}
19/12/10 01:08:05 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/10 01:08:05 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:39609.
19/12/10 01:08:05 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/10 01:08:05 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/10 01:08:05 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 262-1
19/12/10 01:08:05 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:43377.
19/12/10 01:08:05 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/10 01:08:05 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:37915
19/12/10 01:08:05 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/10 01:08:05 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/10 01:08:05 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/10 01:08:05 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/10 01:08:05 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/10 01:08:05 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/10 01:08:05 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/10 01:08:05 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/10 01:08:05 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/10 01:08:05 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/10 01:08:05 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/10 01:08:05 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job test_windowing_1575940079.97_99e63056-ab2b-43ae-97e2-606351b62399 finished.
19/12/10 01:08:05 WARN org.apache.beam.runners.spark.SparkPipelineResult$BatchMode: Collecting monitoring infos is not implemented yet in Spark portable runner.
19/12/10 01:08:05 WARN org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService: Failed to remove job staging directory for token {"sessionId":"job_be1ea735-f5f0-4aa1-aba9-404145ec7f6a","basePath":"/tmp/sparktestjR7oTh"}: {}
java.io.FileNotFoundException: /tmp/sparktestjR7oTh/job_be1ea735-f5f0-4aa1-aba9-404145ec7f6a/MANIFEST (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:118)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:82)
at org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:252)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:88)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService.removeArtifacts(BeamFileSystemArtifactStagingService.java:92)
at org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver.lambda$createJobService$0(JobServerDriver.java:63)
at org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.lambda$run$0(InMemoryJobService.java:201)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.setState(JobInvocation.java:241)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.access$200(JobInvocation.java:48)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:110)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:96)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1058)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE
.
======================================================================
ERROR: test_pardo_state_with_custom_key_coder (__main__.SparkRunnerTest)
Tests that state requests work correctly when the key coder is an
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/portable_runner_test.py", line 231, in test_pardo_state_with_custom_key_coder
equal_to(expected))
File "apache_beam/pipeline.py", line 437, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
======================================================================
ERROR: test_pardo_timers (__main__.SparkRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/fn_api_runner_test.py", line 330, in test_pardo_timers
==================== Timed out after 60 seconds. ====================
assert_that(actual, equal_to(expected))
File "apache_beam/pipeline.py", line 437, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_ru# Thread: <Thread(wait_until_finish_read, started daemon 140280113014528)>
nner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
# Thread: <Thread(Thread-119, started daemon 140280096229120)>
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
# Thread: <_MainThread(MainThread, started 140280892753664)>
==================== Timed out after 60 seconds. ====================
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
# Thread: <Thread(wait_until_finish_read, started daemon 140279470356224)>
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
# Thread: <Thread(Thread-123, started daemon 140280086787840)>
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
# Thread: <_MainThread(MainThread, started 140280892753664)>
======================================================================
ERROR: test_sdf_with_watermark_tracking (__main__.SparkRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
# Thread: <Thread(Thread-119, started daemon 140280096229120)>
File "apache_beam/runners/portability/fn_api_runner_test.py", line 502, in test_sdf_with_watermark_tracking
assert_that(actual, equal_to(list(''.join(data))))
File "apache_beam/pipeline.py", line 437, in __exit__
self.run().wait_until_finish()
# Thread: <Thread(wait_until_finish_read, started daemon 140280113014528)>
File "apache_beam/runners/portability/portable_runner.py", line 438, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline test_sdf_with_watermark_tracking_1575940071.06_db01f780-054e-4e1f-89db-4815b11c31a8 failed in state FAILED: java.lang.UnsupportedOperationException: The ActiveBundle does not have a registered bundle checkpoint handler.
----------------------------------------------------------------------
Ran 38 tests in 319.342s
FAILED (errors=3, skipped=9)
> Task :sdks:python:test-suites:portable:py2:sparkValidatesRunner FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build.gradle'> line: 196
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py2:sparkValidatesRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 7m 58s
60 actionable tasks: 47 executed, 13 from cache
Publishing build scan...
https://scans.gradle.com/s/5fxnkvyk5jluu
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_VR_Spark #1735
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/1735/display/redirect?page=changes>
Changes:
[pabloem] [BEAM-8335] Adds support for multi-output TestStream (#9953)
------------------------------------------
[...truncated 1.55 MB...]
19/12/09 22:32:13 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 22:32:13 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 22:32:14 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 22:32:14 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 22:32:14 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:33585
19/12/09 22:32:14 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 22:32:14 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 22:32:14 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575930732.01_04192864-f9f4-4557-80cb-e9c06487600c', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 22:32:14 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575930732.01', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:50649', 'job_port': u'0'}
19/12/09 22:32:14 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 22:32:14 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:34871.
19/12/09 22:32:14 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 259-1
19/12/09 22:32:14 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 22:32:14 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 22:32:14 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:45863.
19/12/09 22:32:14 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 22:32:14 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:34697
19/12/09 22:32:14 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 22:32:14 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 22:32:14 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 22:32:14 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 22:32:14 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 22:32:14 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 22:32:14 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 22:32:14 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 22:32:14 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 22:32:14 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 22:32:14 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 22:32:14 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 22:32:15 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 22:32:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 22:32:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:40597
19/12/09 22:32:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 22:32:15 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 22:32:15 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575930732.01_04192864-f9f4-4557-80cb-e9c06487600c', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 22:32:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575930732.01', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:50649', 'job_port': u'0'}
19/12/09 22:32:15 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 22:32:15 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:37651.
19/12/09 22:32:15 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 260-1
19/12/09 22:32:15 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 22:32:15 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 22:32:15 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:37431.
19/12/09 22:32:15 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 22:32:15 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:37759
19/12/09 22:32:15 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 22:32:15 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 22:32:15 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 22:32:15 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 22:32:15 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 22:32:15 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 22:32:15 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 22:32:15 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 22:32:15 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 22:32:15 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 22:32:15 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 22:32:15 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 22:32:16 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 22:32:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 22:32:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:40605
19/12/09 22:32:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 22:32:16 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 22:32:16 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575930732.01_04192864-f9f4-4557-80cb-e9c06487600c', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 22:32:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575930732.01', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:50649', 'job_port': u'0'}
19/12/09 22:32:16 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 22:32:16 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:39769.
19/12/09 22:32:16 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 261-1
19/12/09 22:32:16 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 22:32:16 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 22:32:16 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:45149.
19/12/09 22:32:16 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 22:32:16 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:36083
19/12/09 22:32:16 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 22:32:16 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 22:32:16 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 22:32:16 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 22:32:16 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 22:32:16 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 22:32:16 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 22:32:16 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 22:32:16 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 22:32:16 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 22:32:16 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 22:32:16 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 22:32:17 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 22:32:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 22:32:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:38353
19/12/09 22:32:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 22:32:17 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 22:32:17 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575930732.01_04192864-f9f4-4557-80cb-e9c06487600c', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 22:32:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575930732.01', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:50649', 'job_port': u'0'}
19/12/09 22:32:17 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 22:32:17 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:34027.
19/12/09 22:32:17 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 22:32:17 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 22:32:17 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 262-1
19/12/09 22:32:17 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:43745.
19/12/09 22:32:17 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 22:32:17 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:37715
19/12/09 22:32:17 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 22:32:17 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 22:32:17 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 22:32:17 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 22:32:17 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 22:32:17 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 22:32:17 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 22:32:17 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 22:32:17 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 22:32:17 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 22:32:17 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 22:32:17 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job test_windowing_1575930732.01_04192864-f9f4-4557-80cb-e9c06487600c finished.
19/12/09 22:32:17 WARN org.apache.beam.runners.spark.SparkPipelineResult$BatchMode: Collecting monitoring infos is not implemented yet in Spark portable runner.
19/12/09 22:32:17 WARN org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService: Failed to remove job staging directory for token {"sessionId":"job_a0f58f6c-4dba-44df-919e-3ce0502a6998","basePath":"/tmp/sparktest91Ft0Y"}: {}
java.io.FileNotFoundException: /tmp/sparktest91Ft0Y/job_a0f58f6c-4dba-44df-919e-3ce0502a6998/MANIFEST (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:118)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:82)
at org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:252)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:88)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService.removeArtifacts(BeamFileSystemArtifactStagingService.java:92)
at org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver.lambda$createJobService$0(JobServerDriver.java:63)
at org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.lambda$run$0(InMemoryJobService.java:201)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.setState(JobInvocation.java:241)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.access$200(JobInvocation.java:48)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:110)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:96)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1058)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE
.
======================================================================
ERROR: test_pardo_state_with_custom_key_coder (__main__.SparkRunnerTest)
Tests that state requests work correctly when the key coder is an
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/portable_runner_test.py", line 231, in test_pardo_state_with_custom_key_coder
equal_to(expected))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
======================================================================
ERROR: test_pardo_timers (__main__.SparkRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
==================== Timed out after 60 seconds. ====================
File "apache_beam/runners/portability/fn_api_runner_test.py", line 330, in test_pardo_timers
assert_that(actual, equal_to(expected))
# Thread: <Thread(wait_until_finish_read, started daemon 139626339813120)>
# Thread: <Thread(Thread-116, started daemon 139626348205824)>
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
# Thread: <_MainThread(MainThread, started 139627127944960)>
==================== Timed out after 60 seconds. ====================
File "apache_beam/runners/portability/portable_runner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
# Thread: <Thread(wait_until_finish_read, started daemon 139626313848576)>
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
# Thread: <Thread(Thread-122, started daemon 139626322241280)>
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
# Thread: <_MainThread(MainThread, started 139627127944960)>
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
======================================================================
ERROR: test_sdf_with_watermark_tracking (__main__.SparkRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/fn_api_runner_test.py", line 502, in test_sdf_with_watermark_tracking
assert_that(actual, equal_to(list(''.join(data))))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 438, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline test_sdf_with_watermark_tracking_1575930723.54_4186c2fb-8121-47be-a20f-0933e14da306 failed in state FAILED: java.lang.UnsupportedOperationException: The ActiveBundle does not have a registered bundle checkpoint handler.
----------------------------------------------------------------------
Ran 38 tests in 294.635s
FAILED (errors=3, skipped=9)
> Task :sdks:python:test-suites:portable:py2:sparkValidatesRunner FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build.gradle'> line: 196
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py2:sparkValidatesRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 7m 31s
60 actionable tasks: 47 executed, 13 from cache
Publishing build scan...
https://scans.gradle.com/s/ja5j63mpm6jru
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_VR_Spark #1734
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/1734/display/redirect?page=changes>
Changes:
[heejong] [BEAM-8903] handling --jar_packages experimental flag in PortableRunner
------------------------------------------
[...truncated 1.55 MB...]
19/12/09 18:13:37 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:37727
19/12/09 18:13:37 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 18:13:37 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 18:13:37 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575915215.18_baab558b-d6c5-43ba-9edd-ed0bba7fa088', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 18:13:37 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575915215.18', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:58009', 'job_port': u'0'}
19/12/09 18:13:37 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 18:13:37 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:44541.
19/12/09 18:13:37 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 18:13:37 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 259-1
19/12/09 18:13:37 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 18:13:37 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:39105.
19/12/09 18:13:37 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 18:13:37 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:36713
19/12/09 18:13:37 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 18:13:37 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 18:13:37 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 18:13:37 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 18:13:37 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 18:13:37 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 18:13:37 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 18:13:37 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 18:13:37 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 18:13:37 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 18:13:37 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 18:13:37 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 18:13:38 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 18:13:38 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 18:13:38 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:39215
19/12/09 18:13:38 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 18:13:38 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 18:13:38 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575915215.18_baab558b-d6c5-43ba-9edd-ed0bba7fa088', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 18:13:38 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575915215.18', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:58009', 'job_port': u'0'}
19/12/09 18:13:38 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 18:13:38 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:41373.
19/12/09 18:13:38 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 18:13:38 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 18:13:38 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 260-1
19/12/09 18:13:38 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:40849.
19/12/09 18:13:38 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 18:13:38 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:43801
19/12/09 18:13:38 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 18:13:38 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 18:13:38 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 18:13:38 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 18:13:38 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 18:13:38 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 18:13:38 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 18:13:38 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 18:13:38 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 18:13:38 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 18:13:38 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 18:13:38 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 18:13:39 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 18:13:39 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 18:13:39 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:41651
19/12/09 18:13:39 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 18:13:39 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 18:13:39 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575915215.18_baab558b-d6c5-43ba-9edd-ed0bba7fa088', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 18:13:39 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575915215.18', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:58009', 'job_port': u'0'}
19/12/09 18:13:39 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 18:13:39 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:45687.
19/12/09 18:13:39 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 18:13:39 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 18:13:39 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 261-1
19/12/09 18:13:39 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:43385.
19/12/09 18:13:39 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 18:13:39 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:38781
19/12/09 18:13:39 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 18:13:39 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 18:13:39 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 18:13:39 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 18:13:39 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 18:13:39 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 18:13:39 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 18:13:39 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 18:13:39 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 18:13:39 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 18:13:39 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 18:13:39 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 18:13:40 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 18:13:40 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 18:13:40 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:39989
19/12/09 18:13:40 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 18:13:40 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 18:13:40 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575915215.18_baab558b-d6c5-43ba-9edd-ed0bba7fa088', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 18:13:40 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575915215.18', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:58009', 'job_port': u'0'}
19/12/09 18:13:40 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 18:13:40 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:41587.
19/12/09 18:13:40 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 18:13:40 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 18:13:40 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 262-1
19/12/09 18:13:40 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:34507.
19/12/09 18:13:40 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 18:13:40 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:39195
19/12/09 18:13:40 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 18:13:40 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 18:13:40 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 18:13:40 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 18:13:40 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 18:13:40 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 18:13:40 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 18:13:40 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 18:13:40 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 18:13:40 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 18:13:40 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 18:13:40 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job test_windowing_1575915215.18_baab558b-d6c5-43ba-9edd-ed0bba7fa088 finished.
19/12/09 18:13:40 WARN org.apache.beam.runners.spark.SparkPipelineResult$BatchMode: Collecting monitoring infos is not implemented yet in Spark portable runner.
19/12/09 18:13:40 WARN org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService: Failed to remove job staging directory for token {"sessionId":"job_4de5cfcd-0dad-4e22-9d73-f8f0b1992961","basePath":"/tmp/sparktestrNVtlv"}: {}
java.io.FileNotFoundException: /tmp/sparktestrNVtlv/job_4de5cfcd-0dad-4e22-9d73-f8f0b1992961/MANIFEST (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:118)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:82)
at org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:252)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:88)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService.removeArtifacts(BeamFileSystemArtifactStagingService.java:92)
at org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver.lambda$createJobService$0(JobServerDriver.java:63)
at org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.lambda$run$0(InMemoryJobService.java:201)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.setState(JobInvocation.java:241)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.access$200(JobInvocation.java:48)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:110)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:96)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1058)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE
.
======================================================================
ERROR: test_pardo_state_with_custom_key_coder (__main__.SparkRunnerTest)
Tests that state requests work correctly when the key coder is an
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/portable_runner_test.py", line 231, in test_pardo_state_with_custom_key_coder
equal_to(expected))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
==================== Timed out after 60 seconds. ====================
======================================================================
ERROR: test_pardo_timers (__main__.SparkRunnerTest)
----------------------------------------------------------------------
# Thread: <Thread(wait_until_finish_read, started daemon 139728162645760)>
# Thread: <Thread(Thread-120, started daemon 139728154253056)>
# Thread: <_MainThread(MainThread, started 139729287423744)>
==================== Timed out after 60 seconds. ====================
Traceback (most recent call last):
# Thread: <Thread(wait_until_finish_read, started daemon 139728137467648)>
File "apache_beam/runners/portability/fn_api_runner_test.py", line 330, in test_pardo_timers
assert_that(actual, equal_to(expected))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
# Thread: <Thread(Thread-124, started daemon 139728145860352)>
# Thread: <Thread(Thread-120, started daemon 139728154253056)>
File "apache_beam/runners/portability/portable_ru# Thread: <Thread(wait_until_finish_read, started daemon 139728162645760)>
nner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
# Thread: <_MainThread(MainThread, started 139729287423744)>
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
======================================================================
ERROR: test_sdf_with_watermark_tracking (__main__.SparkRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/fn_api_runner_test.py", line 502, in test_sdf_with_watermark_tracking
assert_that(actual, equal_to(list(''.join(data))))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 438, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline test_sdf_with_watermark_tracking_1575915207.26_5894b3c6-6fb0-4395-9732-c8e5700a2208 failed in state FAILED: java.lang.UnsupportedOperationException: The ActiveBundle does not have a registered bundle checkpoint handler.
----------------------------------------------------------------------
Ran 38 tests in 344.483s
FAILED (errors=3, skipped=9)
> Task :sdks:python:test-suites:portable:py2:sparkValidatesRunner FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build.gradle'> line: 196
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py2:sparkValidatesRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 20s
60 actionable tasks: 47 executed, 13 from cache
Publishing build scan...
https://scans.gradle.com/s/wmzs676h6pnly
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_VR_Spark #1733
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/1733/display/redirect?page=changes>
Changes:
[dcavazos] [BEAM-7390] Add code snippet for Mean
[nielm] Add limit on number of mutated rows to batching/sorting stages.
------------------------------------------
[...truncated 1.54 MB...]
19/12/09 17:50:25 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Running job test_windowing_1575913824.26_55dcbf41-75bc-4cd6-90c1-110dc0bb998e on Spark master local
19/12/09 17:50:25 WARN org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions: Either coder LengthPrefixCoder(ByteArrayCoder) or GlobalWindow$Coder is not consistent with equals. That might cause issues on some runners.
19/12/09 17:50:25 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job test_windowing_1575913824.26_55dcbf41-75bc-4cd6-90c1-110dc0bb998e: Pipeline translated successfully. Computing outputs
19/12/09 17:50:25 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 17:50:25 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 17:50:25 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 17:50:25 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:37601
19/12/09 17:50:25 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 17:50:25 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 17:50:25 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575913824.26_55dcbf41-75bc-4cd6-90c1-110dc0bb998e', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 17:50:25 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575913824.26', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:44153', 'job_port': u'0'}
19/12/09 17:50:25 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 17:50:25 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:45333.
19/12/09 17:50:25 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 17:50:25 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 17:50:25 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 258-1
19/12/09 17:50:25 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:34645.
19/12/09 17:50:25 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 17:50:25 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:44701
19/12/09 17:50:25 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 17:50:25 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 17:50:25 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 17:50:25 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 17:50:25 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:25 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 17:50:25 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 17:50:25 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 17:50:25 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 17:50:25 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 17:50:26 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:26 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 17:50:26 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 17:50:26 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 17:50:26 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:42917
19/12/09 17:50:26 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 17:50:26 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 17:50:26 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575913824.26_55dcbf41-75bc-4cd6-90c1-110dc0bb998e', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 17:50:26 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575913824.26', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:44153', 'job_port': u'0'}
19/12/09 17:50:26 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 17:50:26 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:38947.
19/12/09 17:50:26 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 259-1
19/12/09 17:50:26 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 17:50:26 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 17:50:26 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:42731.
19/12/09 17:50:26 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 17:50:26 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:43735
19/12/09 17:50:26 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 17:50:26 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 17:50:26 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 17:50:26 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 17:50:26 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:26 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 17:50:26 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 17:50:26 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 17:50:26 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 17:50:26 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 17:50:26 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:26 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 17:50:27 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 17:50:27 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 17:50:27 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:36873
19/12/09 17:50:27 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 17:50:27 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 17:50:27 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575913824.26_55dcbf41-75bc-4cd6-90c1-110dc0bb998e', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 17:50:27 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575913824.26', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:44153', 'job_port': u'0'}
19/12/09 17:50:27 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 17:50:27 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:35647.
19/12/09 17:50:27 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 17:50:27 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 260-1
19/12/09 17:50:27 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 17:50:27 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:39271.
19/12/09 17:50:27 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 17:50:27 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 17:50:27 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:42263
19/12/09 17:50:27 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 17:50:27 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 17:50:27 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 17:50:27 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:27 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 17:50:27 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 17:50:27 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 17:50:27 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 17:50:27 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 17:50:27 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:27 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 17:50:28 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 17:50:28 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 17:50:28 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:39309
19/12/09 17:50:28 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 17:50:28 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 17:50:28 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575913824.26_55dcbf41-75bc-4cd6-90c1-110dc0bb998e', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 17:50:28 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575913824.26', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:44153', 'job_port': u'0'}
19/12/09 17:50:28 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 17:50:28 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:44753.
19/12/09 17:50:28 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 261-1
19/12/09 17:50:28 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 17:50:28 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 17:50:28 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:45279.
19/12/09 17:50:28 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 17:50:28 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:42871
19/12/09 17:50:28 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 17:50:28 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 17:50:28 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 17:50:28 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 17:50:28 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 17:50:28 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:28 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 17:50:28 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 17:50:28 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 17:50:28 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 17:50:28 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:28 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 17:50:29 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 17:50:29 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 17:50:29 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:44081
19/12/09 17:50:29 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 17:50:29 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 17:50:29 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575913824.26_55dcbf41-75bc-4cd6-90c1-110dc0bb998e', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 17:50:29 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575913824.26', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:44153', 'job_port': u'0'}
19/12/09 17:50:29 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 17:50:29 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:45375.
19/12/09 17:50:29 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 17:50:29 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 262-1
19/12/09 17:50:29 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 17:50:29 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:45793.
19/12/09 17:50:29 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 17:50:29 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:35891
19/12/09 17:50:29 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 17:50:29 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 17:50:29 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 17:50:29 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 17:50:29 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 17:50:29 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 17:50:29 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 17:50:29 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:29 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 17:50:29 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 17:50:29 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 17:50:29 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job test_windowing_1575913824.26_55dcbf41-75bc-4cd6-90c1-110dc0bb998e finished.
19/12/09 17:50:29 WARN org.apache.beam.runners.spark.SparkPipelineResult$BatchMode: Collecting monitoring infos is not implemented yet in Spark portable runner.
19/12/09 17:50:29 WARN org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService: Failed to remove job staging directory for token {"sessionId":"job_611a5cae-20e5-408f-a1d5-b36b36a8a2a2","basePath":"/tmp/sparktestKn42fl"}: {}
java.io.FileNotFoundException: /tmp/sparktestKn42fl/job_611a5cae-20e5-408f-a1d5-b36b36a8a2a2/MANIFEST (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:118)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:82)
at org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:252)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:88)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService.removeArtifacts(BeamFileSystemArtifactStagingService.java:92)
at org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver.lambda$createJobService$0(JobServerDriver.java:63)
at org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.lambda$run$0(InMemoryJobService.java:201)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.setState(JobInvocation.java:241)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.access$200(JobInvocation.java:48)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:110)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:96)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1058)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE
.
======================================================================
ERROR: test_pardo_state_with_custom_key_coder (__main__.SparkRunnerTest)
Tests that state requests work correctly when the key coder is an
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/portable_runner_test.py", line 231, in test_pardo_state_with_custom_key_coder
equal_to(expected))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
======================================================================
ERROR: test_sdf_with_watermark_tracking (__main__.SparkRunnerTest)
----------------------------------------------------------------------
==================== Timed out after 60 seconds. ====================
Traceback (most recent call last):
File "apache_beam/runners/portability/fn_api_runner_test.py", line 502, in test_sdf_with_watermark_tracking
assert_that(actual, equal_to(list(''.join(data))))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 438, in wait_until_finish
# Thread: <Thread(wait_until_finish_read, started daemon 140431663650560)>
self._job_id, self._state, self._last_error_message()))
# Thread: <Thread(Thread-119, started daemon 140431655257856)>
RuntimeError: Pipeline test_sdf_with_watermark_tracking_1575913816.41_e7496de5-5756-4596-b053-c4a306ceff8b failed in state FAILED: java.lang.UnsupportedOperationException: The ActiveBundle does not have a registered bundle checkpoint handler.
----------------------------------------------------------------------
# Thread: <_MainThread(MainThread, started 140432443389696)>
Ran 38 tests in 263.596s
FAILED (errors=2, skipped=9)
> Task :sdks:python:test-suites:portable:py2:sparkValidatesRunner FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build.gradle'> line: 196
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py2:sparkValidatesRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 6m 55s
60 actionable tasks: 47 executed, 13 from cache
Publishing build scan...
https://scans.gradle.com/s/lb743pq37khum
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python_VR_Spark #1732
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/1732/display/redirect?page=changes>
Changes:
[github] Changing RowAsDictJsonCoder implementation for efficiency (#10300)
[github] Merge pull request #10151: [BEAM-7116] Remove use of KV in Schema
------------------------------------------
[...truncated 1.54 MB...]
19/12/09 16:58:31 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Running job test_windowing_1575910710.89_cf097cfe-16f5-42ec-8002-4d9af9623a9f on Spark master local
19/12/09 16:58:31 WARN org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions: Either coder LengthPrefixCoder(ByteArrayCoder) or GlobalWindow$Coder is not consistent with equals. That might cause issues on some runners.
19/12/09 16:58:31 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job test_windowing_1575910710.89_cf097cfe-16f5-42ec-8002-4d9af9623a9f: Pipeline translated successfully. Computing outputs
19/12/09 16:58:31 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 16:58:33 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 16:58:33 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 16:58:33 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:40919
19/12/09 16:58:33 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 16:58:33 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 16:58:33 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575910710.89_cf097cfe-16f5-42ec-8002-4d9af9623a9f', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 16:58:33 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575910710.89', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:40169', 'job_port': u'0'}
19/12/09 16:58:33 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 16:58:33 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:36993.
19/12/09 16:58:33 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 16:58:33 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 258-1
19/12/09 16:58:33 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 16:58:33 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:39161.
19/12/09 16:58:33 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 16:58:33 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:46663
19/12/09 16:58:33 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 16:58:33 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 16:58:33 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 16:58:33 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 16:58:33 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 16:58:33 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 16:58:33 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:33 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 16:58:33 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 16:58:33 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 16:58:33 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:33 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 16:58:33 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 16:58:33 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 16:58:33 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:37491
19/12/09 16:58:34 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 16:58:34 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 16:58:34 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575910710.89_cf097cfe-16f5-42ec-8002-4d9af9623a9f', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 16:58:34 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575910710.89', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:40169', 'job_port': u'0'}
19/12/09 16:58:34 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:33405.
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 259-1
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:41857.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 16:58:34 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:46185
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 16:58:34 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 16:58:34 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 16:58:34 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 16:58:34 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 16:58:34 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 16:58:34 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:35791
19/12/09 16:58:34 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 16:58:34 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 16:58:34 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575910710.89_cf097cfe-16f5-42ec-8002-4d9af9623a9f', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 16:58:34 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575910710.89', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:40169', 'job_port': u'0'}
19/12/09 16:58:34 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:38381.
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 260-1
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:44513.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 16:58:34 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:38493
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 16:58:34 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:34 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 16:58:34 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 16:58:34 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 16:58:34 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 16:58:35 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:35 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 16:58:35 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 16:58:35 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 16:58:35 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:39593
19/12/09 16:58:35 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 16:58:35 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 16:58:35 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575910710.89_cf097cfe-16f5-42ec-8002-4d9af9623a9f', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 16:58:35 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575910710.89', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:40169', 'job_port': u'0'}
19/12/09 16:58:35 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 16:58:35 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:44567.
19/12/09 16:58:35 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 16:58:35 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 16:58:35 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 261-1
19/12/09 16:58:35 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:35541.
19/12/09 16:58:35 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 16:58:35 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:41561
19/12/09 16:58:35 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 16:58:35 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 16:58:35 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 16:58:35 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 16:58:35 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:35 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 16:58:35 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 16:58:35 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 16:58:35 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 16:58:35 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 16:58:35 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:35 INFO org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService: GetManifest for __no_artifacts_staged__
19/12/09 16:58:36 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging client connected.
19/12/09 16:58:36 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:106: Logging handler created.
19/12/09 16:58:36 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:88: Status HTTP server running at localhost:33543
19/12/09 16:58:36 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:132: semi_persistent_directory: /tmp
19/12/09 16:58:36 WARN <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:213: No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
19/12/09 16:58:36 WARN apache_beam/options/pipeline_options.py:268: Discarding unparseable args: [u'--app_name=test_windowing_1575910710.89_cf097cfe-16f5-42ec-8002-4d9af9623a9f', u'--job_server_timeout=60', u'--pipeline_type_check', u'--direct_runner_use_stacked_bundle', u'--spark_master=local', u'--options_id=30', u'--enable_spark_metric_sinks']
19/12/09 16:58:36 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:144: Python sdk harness started with pipeline_options: {'runner': u'None', 'experiments': [u'beam_fn_api'], 'environment_cache_millis': u'0', 'artifact_port': u'0', 'environment_type': u'PROCESS', 'sdk_location': u'container', 'job_name': u'test_windowing_1575910710.89', 'environment_config': u'{"command": "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',> 'expansion_port': u'0', 'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:40169', 'job_port': u'0'}
19/12/09 16:58:36 INFO apache_beam/runners/worker/statecache.py:137: Creating state cache with size 0
19/12/09 16:58:36 INFO apache_beam/runners/worker/sdk_worker.py:71: Creating insecure control channel for localhost:45585.
19/12/09 16:58:36 INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam Fn Control client connected with id 262-1
19/12/09 16:58:36 INFO apache_beam/runners/worker/sdk_worker.py:79: Control channel established.
19/12/09 16:58:36 INFO apache_beam/runners/worker/sdk_worker.py:96: Initializing SDKHarness with unbounded number of workers.
19/12/09 16:58:36 INFO apache_beam/runners/worker/sdk_worker.py:449: Creating insecure state channel for localhost:36269.
19/12/09 16:58:36 INFO apache_beam/runners/worker/sdk_worker.py:456: State channel established.
19/12/09 16:58:36 INFO apache_beam/runners/worker/data_plane.py:354: Creating client data channel for localhost:36271
19/12/09 16:58:36 INFO org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client connected.
19/12/09 16:58:36 INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing environment urn: "beam:env:process:v1"
payload: "\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh">
19/12/09 16:58:36 INFO apache_beam/runners/worker/sdk_worker.py:122: No more requests from control plane
19/12/09 16:58:36 INFO apache_beam/runners/worker/sdk_worker.py:123: SDK Harness waiting for in-flight requests to complete
19/12/09 16:58:36 INFO apache_beam/runners/worker/data_plane.py:376: Closing all cached grpc data channels.
19/12/09 16:58:36 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:36 INFO apache_beam/runners/worker/sdk_worker.py:467: Closing all cached gRPC state handlers.
19/12/09 16:58:36 INFO apache_beam/runners/worker/sdk_worker.py:133: Done consuming work.
19/12/09 16:58:36 INFO <https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker_main.py>:157: Python sdk harness exiting.
19/12/09 16:58:36 INFO org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Logging client hanged up.
19/12/09 16:58:36 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer: Hanged up for unknown endpoint.
19/12/09 16:58:36 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job test_windowing_1575910710.89_cf097cfe-16f5-42ec-8002-4d9af9623a9f finished.
19/12/09 16:58:36 WARN org.apache.beam.runners.spark.SparkPipelineResult$BatchMode: Collecting monitoring infos is not implemented yet in Spark portable runner.
19/12/09 16:58:36 WARN org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService: Failed to remove job staging directory for token {"sessionId":"job_26fbcb58-6bf1-428b-9821-76f67e85271a","basePath":"/tmp/sparktestYC2f7d"}: {}
java.io.FileNotFoundException: /tmp/sparktestYC2f7d/job_26fbcb58-6bf1-428b-9821-76f67e85271a/MANIFEST (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:118)
at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:82)
at org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:252)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:88)
at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService.removeArtifacts(BeamFileSystemArtifactStagingService.java:92)
at org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver.lambda$createJobService$0(JobServerDriver.java:63)
at org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.lambda$run$0(InMemoryJobService.java:201)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.setState(JobInvocation.java:241)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.access$200(JobInvocation.java:48)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:110)
at org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation$1.onSuccess(JobInvocation.java:96)
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1058)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE
.
======================================================================
ERROR: test_pardo_state_with_custom_key_coder (__main__.SparkRunnerTest)
Tests that state requests work correctly when the key coder is an
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/portable_runner_test.py", line 231, in test_pardo_state_with_custom_key_coder
equal_to(expected))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 428, in wait_until_finish
for state_response in self._state_stream:
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 395, in next
return self._next()
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py",> line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File "<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py",> line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 75, in handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
======================================================================
ERROR: test_sdf_with_watermark_tracking (__main__.SparkRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/fn_api_runner_test.py", line 502, in test_sdf_with_watermark_tracking
assert_that(actual, equal_to(list(''.join(data))))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 438, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
==================== Timed out after 60 seconds. ====================
RuntimeError: Pipeline test_sdf_with_watermark_tracking_1575910703.23_f383ba58-4745-4b22-ac38-2d7365a9bd69 failed in state FAILED: java.lang.UnsupportedOperationException: The ActiveBundle does not have a registered bundle checkpoint handler.
# Thread: <Thread(wait_until_finish_read, started daemon 140595631769344)>
# Thread: <Thread(Thread-120, started daemon 140595284670208)>
----------------------------------------------------------------------
Ran 38 tests in 286.679s
# Thread: <_MainThread(MainThread, started 140596411008768)>
FAILED (errors=2, skipped=9)
> Task :sdks:python:test-suites:portable:py2:sparkValidatesRunner FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build.gradle'> line: 196
* What went wrong:
Execution failed for task ':sdks:python:test-suites:portable:py2:sparkValidatesRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 7m 19s
60 actionable tasks: 56 executed, 4 from cache
Publishing build scan...
https://scans.gradle.com/s/jzxwv7uqdgftq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org