You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Ștefan Istrate (Jira)" <ji...@apache.org> on 2021/07/26 11:37:00 UTC
[jira] [Updated] (BEAM-12657) Can't run Beam on separate Flink
cluster. Official instructions don't work.
[ https://issues.apache.org/jira/browse/BEAM-12657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ștefan Istrate updated BEAM-12657:
----------------------------------
Description:
I am trying to run Beam on a separate Flink cluster, as described here: [https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster]
??(1) Start a Flink cluster which exposes the Rest interface (e.g. {{localhost:8081}} by default).??
Flink 1.13.1 starts successfully on localhost:8081.
??(2) Start JobService with Flink Rest endpoint: {{docker run --net=host apache/beam_flink1.10_job_server:latest --flink-master=localhost:8081}}.??
I am running this instead, which again starts successfully:
{{docker run --net=host apache/beam_flink1.13_job_server:latest --flink-master=localhost:8081}}
??(3) Submit the pipeline as above.??
I'm running the WordCount pipeline as:
{{python -m apache_beam.examples.wordcount --input /Users/stefan/datastore/input.txt}}
{{ --output /Users/stefan/datastore/output.txt}}
{{ --runner=PortableRunner --job_endpoint=localhost:8099}}
After 1 minute the pipeline crashes with a `grpc.FutureTimeoutError`. This is the full output:
{{WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.}}
{{INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0}}
{{INFO:root:No image given, using default Python SDK image}}
{{WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.}}
{{INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0}}
{{INFO:root:Python SDK container image set to "apache/beam_python3.8_sdk:2.31.0" for Docker environment}}
{{INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x137d441f0> ====================}}
{{INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x137d44280> ====================}}
{{INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x137d449d0> ====================}}
{{Traceback (most recent call last):}}
\{{ File "/usr/local/Cellar/python@3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main}}
\{{ return _run_code(code, main_globals, None,}}
\{{ File "/usr/local/Cellar/python@3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code}}
\{{ exec(code, run_globals)}}
\{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 94, in <module>}}
\{{ run()}}
\{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 89, in run}}
\{{ output | 'Write' >> WriteToText(known_args.output)}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 585, in __exit__}}
\{{ self.result = self.run()}}
\{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 564, in run}}
\{{ return self.runner.run_pipeline(self, self._options)}}
\{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 438, in run_pipeline}}
\{{ job_service_handle = self.create_job_service(options)}}
\{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 317, in create_job_service}}
\{{ return self.create_job_service_handle(server.start(), options)}}
\{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/job_server.py", line 54, in start}}
\{{ grpc.channel_ready_future(channel).result(timeout=self._timeout)}}
\{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 140, in result}}
\{{ self._block(timeout)}}
\{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 86, in _block}}
\{{ raise grpc.FutureTimeoutError()}}
{{grpc.FutureTimeoutError}}
What's going on? Any help would be appreciated.
was:
I am trying to run Beam on a separate Flink cluster, as described here: [https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster]
??(1) Start a Flink cluster which exposes the Rest interface (e.g. {{localhost:8081}} by default).??
Flink 1.13.1 starts successfully on localhost:8081.
??(2) Start JobService with Flink Rest endpoint: {{docker run --net=host apache/beam_flink1.10_job_server:latest --flink-master=localhost:8081}}.??
I am running this instead, which again starts successfully:
{{docker run --net=host apache/beam_flink1.13_job_server:latest --flink-master=localhost:8081}}
??(3) Submit the pipeline as above.??
I'm running the WordCount pipeline as:
{{python -m apache_beam.examples.wordcount --input /Users/stefan/datastore/input.txt \}}
{{ --output /Users/stefan/datastore/output.txt \}}
{{ --runner=PortableRunner --job_endpoint=localhost:8099}}
After 1 minute the pipeline crashes with a `grpc.FutureTimeoutError`. This is the full output:
{{WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.}}
{{INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0}}
{{INFO:root:No image given, using default Python SDK image}}
{{WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.}}
{{INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0}}
{{INFO:root:Python SDK container image set to "apache/beam_python3.8_sdk:2.31.0" for Docker environment}}
{{INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x137d441f0> ====================}}
{{INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x137d44280> ====================}}
{{INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x137d449d0> ====================}}
{{Traceback (most recent call last):}}
{{ File "/usr/local/Cellar/python@3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main}}
{{ return _run_code(code, main_globals, None,}}
{{ File "/usr/local/Cellar/python@3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code}}
{{ exec(code, run_globals)}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 94, in <module>}}
{{ run()}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 89, in run}}
{{ output | 'Write' >> WriteToText(known_args.output)}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 585, in __exit__}}
{{ self.result = self.run()}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 564, in run}}
{{ return self.runner.run_pipeline(self, self._options)}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 438, in run_pipeline}}
{{ job_service_handle = self.create_job_service(options)}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 317, in create_job_service}}
{{ return self.create_job_service_handle(server.start(), options)}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/job_server.py", line 54, in start}}
{{ grpc.channel_ready_future(channel).result(timeout=self._timeout)}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 140, in result}}
{{ self._block(timeout)}}
{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 86, in _block}}
{{ raise grpc.FutureTimeoutError()}}
{{grpc.FutureTimeoutError}}
What's going on? Any help would be appreciated.
> Can't run Beam on separate Flink cluster. Official instructions don't work.
> ---------------------------------------------------------------------------
>
> Key: BEAM-12657
> URL: https://issues.apache.org/jira/browse/BEAM-12657
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Affects Versions: 2.31.0
> Reporter: Ștefan Istrate
> Priority: P2
>
> I am trying to run Beam on a separate Flink cluster, as described here: [https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster]
> ??(1) Start a Flink cluster which exposes the Rest interface (e.g. {{localhost:8081}} by default).??
> Flink 1.13.1 starts successfully on localhost:8081.
> ??(2) Start JobService with Flink Rest endpoint: {{docker run --net=host apache/beam_flink1.10_job_server:latest --flink-master=localhost:8081}}.??
> I am running this instead, which again starts successfully:
> {{docker run --net=host apache/beam_flink1.13_job_server:latest --flink-master=localhost:8081}}
> ??(3) Submit the pipeline as above.??
> I'm running the WordCount pipeline as:
> {{python -m apache_beam.examples.wordcount --input /Users/stefan/datastore/input.txt}}
> {{ --output /Users/stefan/datastore/output.txt}}
> {{ --runner=PortableRunner --job_endpoint=localhost:8099}}
> After 1 minute the pipeline crashes with a `grpc.FutureTimeoutError`. This is the full output:
>
> {{WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.}}
> {{INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0}}
> {{INFO:root:No image given, using default Python SDK image}}
> {{WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.}}
> {{INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.31.0}}
> {{INFO:root:Python SDK container image set to "apache/beam_python3.8_sdk:2.31.0" for Docker environment}}
> {{INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x137d441f0> ====================}}
> {{INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x137d44280> ====================}}
> {{INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x137d449d0> ====================}}
> {{Traceback (most recent call last):}}
> \{{ File "/usr/local/Cellar/python@3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main}}
> \{{ return _run_code(code, main_globals, None,}}
> \{{ File "/usr/local/Cellar/python@3.8/3.8.11/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code}}
> \{{ exec(code, run_globals)}}
> \{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 94, in <module>}}
> \{{ run()}}
> \{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/examples/wordcount.py", line 89, in run}}
> \{{ output | 'Write' >> WriteToText(known_args.output)}}
> {{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 585, in __exit__}}
> \{{ self.result = self.run()}}
> \{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/pipeline.py", line 564, in run}}
> \{{ return self.runner.run_pipeline(self, self._options)}}
> \{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 438, in run_pipeline}}
> \{{ job_service_handle = self.create_job_service(options)}}
> \{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/portable_runner.py", line 317, in create_job_service}}
> \{{ return self.create_job_service_handle(server.start(), options)}}
> \{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/apache_beam/runners/portability/job_server.py", line 54, in start}}
> \{{ grpc.channel_ready_future(channel).result(timeout=self._timeout)}}
> \{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 140, in result}}
> \{{ self._block(timeout)}}
> \{{ File "/Users/stefan/workspace/biodiversity/.env/lib/python3.8/site-packages/grpc/_utilities.py", line 86, in _block}}
> \{{ raise grpc.FutureTimeoutError()}}
> {{grpc.FutureTimeoutError}}
>
> What's going on? Any help would be appreciated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)