You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Xander Song <ia...@gmail.com> on 2020/02/07 04:15:49 UTC
Running Beam on Flink
I am having difficulty following the Python guide for running Beam on Flink
<https://beam.apache.org/documentation/runners/flink/>. I created a virtual
environment with Apache Beam installed, then I started up the JobService
Docker container with
docker run --net=host apachebeam/flink1.9_job_server:latest
I receive the following message confirming that the container is running.
[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
ArtifactStagingService started on localhost:8098
[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
ExpansionService started on localhost:8097
[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
JobService started on localhost:8099
In another terminal, I execute a Beam script called
test_beam_local_flink.py based on the example.
from __future__ import print_function
import apache_beamfrom apache_beam.options.pipeline_options import
PipelineOptions
data = [1,2,3]
options = PipelineOptions([
"--runner=PortableRunner",
"--job_endpoint=localhost:8099",
"--environment_type=LOOPBACK"])
with apache_beam.Pipeline(options=options) as p:
video_collection = (
p | apache_beam.Create(data)
| apache_beam.Map(lambda x: x + 1)
| apache_beam.Map(lambda x: print(x))
)
print('Done')
After a wait, I get the following traceback.
/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84:
UserWarning: You are using Apache Beam with Python 2. New releases of
Apache Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
Traceback (most recent call last):
File "test_beam_local_flink.py", line 18, in <module>
| apache_beam.Map(lambda x: print(x))
File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 481, in __exit__
self.run().wait_until_finish()
File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 461, in run
self._options).run(False)
File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 474, in run
return self.runner.run_pipeline(self, self._options)
File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 220, in run_pipeline
job_service = self.create_job_service(options)
File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 136, in create_job_service
return server.start()
File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py",
line 59, in start
grpc.channel_ready_future(channel).result(timeout=self._timeout)
File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
line 140, in result
self._block(timeout)
File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
line 86, in _block
raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError
Any help is greatly appreciated.
Re: Running Beam on Flink
Posted by Yu Watanabe <yu...@gmail.com>.
Hello .
From this line in the log, may I confirm whether you set up harness container ?
I have executed your code in below environment and it worked. Python
version is newer though.
OS: Fedora 31
BEAM: 2.19.0
PYTHON: 3.7.6
Compared to my log , I realized below line does not exist in your log.
--------------------------------------------------------------------------------------------
357 [grpc-default-executor-1] INFO
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService
- Beam Fn Control client connected with id 1-1
358 [grpc-default-executor-1] INFO
org.apache.beam.runners.fnexecution.data.GrpcDataService - Beam Fn
Data client connected.
--------------------------------------------------------------------------------------------
One thing I realized was in LOOPBACK mode for harness container , job
server uses hostname for when communicating with flink.
_______________________________________________________
Caused by: java.net.UnknownHostException: fedora-desktop: Name or
service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
... 34 more
_______________________________________________________
Perhaps temporarily adding your hostname to "hosts file" by resolving
with 127.0.0.1 and see if it works ?
Thanks,
Yu Watanabe
On Mon, Feb 10, 2020 at 3:29 AM Xander Song <ia...@gmail.com> wrote:
>
> Hi Jincheng,
>
> Thanks for your help. Yes, I am using Mac. Your suggestion allowed me to submit the job on port 8099. However, I am now encountering a different error message.
>
> ERROR:root:java.net.ConnectException: Connection refused
>
> Traceback (most recent call last):
>
> File "test_beam_local_flink.py", line 18, in <module>
>
> | apache_beam.Map(lambda x: print(x))
>
> File "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py", line 481, in __exit__
>
> self.run().wait_until_finish()
>
> File "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py", line 461, in run
>
> self._options).run(False)
>
> File "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py", line 474, in run
>
> return self.runner.run_pipeline(self, self._options)
>
> File "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 334, in run_pipeline
>
> result.wait_until_finish()
>
> File "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 455, in wait_until_finish
>
> self._job_id, self._state, self._last_error_message()))
>
> RuntimeError: Pipeline BeamApp-root-0209182021-fc6ec677_c351e1bb-d428-4475-9910-4544d8ec1de4 failed in state FAILED: java.net.ConnectException: Connection refused
>
>
> I've attached a text file containing the output from the terminal running the Docker container in case that is informative (it is quite lengthy). Any suggestions are appreciated.
>
> Best,
> Xander
>
> On Sun, Feb 9, 2020 at 2:44 AM jincheng sun <su...@gmail.com> wrote:
>>
>>
>> Hi Xander,
>>
>> Are you using Mac? The option --net=host doesn't work on Mac[1]. Could you try to see if the command
>> `docker run -p 8099:8099 -p 8098:8098 -p 8097:8097 apachebeam/flink1.9_job_server:latest` works?
>>
>> [1] https://forums.docker.com/t/should-docker-run-net-host-work/14215/28
>>
>> Best,
>> Jincheng
>> -------------
>> Twitter: https://twitter.com/sunjincheng121
>> -------------
>>
>>
>> Xander Song <ia...@gmail.com> 于2020年2月9日周日 上午11:52写道：
>>>
>>> Do you have any suggestions for addressing this issue? I am unsure of what to try next.
>>>
>>> On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka <go...@google.com> wrote:
>>>>
>>>> That seems to be a problem.
>>>>
>>>> When I try the command, I get
>>>>
>>>> $ telnet localhost 8099
>>>> Trying ::1...
>>>> Connected to localhost.
>>>> Escape character is '^]'.
>>>> �^CConnection closed by foreign host.
>>>>
>>>> On Fri, Feb 7, 2020 at 5:34 PM Xander Song <ia...@gmail.com> wrote:
>>>>>
>>>>> Thanks for the response. After entering telnet localhost 8099, I receive
>>>>>
>>>>> Trying ::1...
>>>>>
>>>>> telnet: connect to address ::1: Connection refused
>>>>>
>>>>> Trying 127.0.0.1...
>>>>>
>>>>> telnet: connect to address 127.0.0.1: Connection refused
>>>>>
>>>>> telnet: Unable to connect to remote host
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka <go...@google.com> wrote:
>>>>>>
>>>>>> Seems that pipeline submission from sdk is not able to reach the job server which was started in docker.
>>>>>>
>>>>>> Can you try running "telnet localhost 8099" to make sure that pipeline submission can reach the job server.
>>>>>>
>>>>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song <ia...@gmail.com> wrote:
>>>>>>>
>>>>>>> I am having difficulty following the Python guide for running Beam on Flink. I created a virtual environment with Apache Beam installed, then I started up the JobService Docker container with
>>>>>>>
>>>>>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>>>>>
>>>>>>>
>>>>>>> I receive the following message confirming that the container is running.
>>>>>>>
>>>>>>>
>>>>>>> [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - ArtifactStagingService started on localhost:8098
>>>>>>>
>>>>>>> [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java ExpansionService started on localhost:8097
>>>>>>>
>>>>>>> [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - JobService started on localhost:8099
>>>>>>>
>>>>>>>
>>>>>>> In another terminal, I execute a Beam script called test_beam_local_flink.py based on the example.
>>>>>>>
>>>>>>>
>>>>>>> from __future__ import print_function
>>>>>>>
>>>>>>> import apache_beam
>>>>>>> from apache_beam.options.pipeline_options import PipelineOptions
>>>>>>>
>>>>>>> data = [1,2,3]
>>>>>>>
>>>>>>> options = PipelineOptions([
>>>>>>> "--runner=PortableRunner",
>>>>>>> "--job_endpoint=localhost:8099",
>>>>>>> "--environment_type=LOOPBACK"
>>>>>>> ])
>>>>>>>
>>>>>>> with apache_beam.Pipeline(options=options) as p:
>>>>>>> video_collection = (
>>>>>>> p | apache_beam.Create(data)
>>>>>>> | apache_beam.Map(lambda x: x + 1)
>>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>>> )
>>>>>>>
>>>>>>> print('Done')
>>>>>>>
>>>>>>> After a wait, I get the following traceback.
>>>>>>>
>>>>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only.
>>>>>>>
>>>>>>> 'You are using Apache Beam with Python 2. '
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>>
>>>>>>> File "test_beam_local_flink.py", line 18, in <module>
>>>>>>>
>>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 481, in __exit__
>>>>>>>
>>>>>>> self.run().wait_until_finish()
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 461, in run
>>>>>>>
>>>>>>> self._options).run(False)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 474, in run
>>>>>>>
>>>>>>> return self.runner.run_pipeline(self, self._options)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 220, in run_pipeline
>>>>>>>
>>>>>>> job_service = self.create_job_service(options)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 136, in create_job_service
>>>>>>>
>>>>>>> return server.start()
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py", line 59, in start
>>>>>>>
>>>>>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 140, in result
>>>>>>>
>>>>>>> self._block(timeout)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 86, in _block
>>>>>>>
>>>>>>> raise grpc.FutureTimeoutError()
>>>>>>>
>>>>>>> grpc.FutureTimeoutError
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any help is greatly appreciated.
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: Running Beam on Flink
Posted by jincheng sun <su...@gmail.com>.
Hi Xander,
It will start an external Python worker pool service at the client side
when submitting the job if the environment type is configured as
"LOOPBACK". The operators(which run in docker) of the job will try to
connect to the external Python worker pool service to start the Python SDK
harness. The reason of the latest failure is because the operators which
run in the docker could not connect to the service which is outside of the
docker container(as a result it could not connect to the external Python
worker pool service). I'm sorry that I have also no idea about the best
practise for how to make it works if the job service runs in a docker
container in Mac. Before knowing of the best practise, I guess you could
start up the job service directly on the bare metal.
Best,
Jincheng
-------------
Twitter: https://twitter.com/sunjincheng121
-------------
Xander Song <ia...@gmail.com> 于2020年2月10日周一 上午2:29写道：
> Hi Jincheng,
>
> Thanks for your help. Yes, I am using Mac. Your suggestion allowed me to
> submit the job on port 8099. However, I am now encountering a different
> error message.
>
> ERROR:root:java.net.ConnectException: Connection refused
>
> Traceback (most recent call last):
>
> File "test_beam_local_flink.py", line 18, in <module>
>
> | apache_beam.Map(lambda x: print(x))
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py",
> line 481, in __exit__
>
> self.run().wait_until_finish()
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py",
> line 461, in run
>
> self._options).run(False)
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py",
> line 474, in run
>
> return self.runner.run_pipeline(self, self._options)
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 334, in run_pipeline
>
> result.wait_until_finish()
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 455, in wait_until_finish
>
> self._job_id, self._state, self._last_error_message()))
>
> RuntimeError: Pipeline
> BeamApp-root-0209182021-fc6ec677_c351e1bb-d428-4475-9910-4544d8ec1de4
> failed in state FAILED: java.net.ConnectException: Connection refused
>
> I've attached a text file containing the output from the terminal running
> the Docker container in case that is informative (it is quite lengthy). Any
> suggestions are appreciated.
>
> Best,
> Xander
>
> On Sun, Feb 9, 2020 at 2:44 AM jincheng sun <su...@gmail.com>
> wrote:
>
>>
>> Hi Xander,
>>
>> Are you using Mac? The option --net=host doesn't work on Mac[1]. Could
>> you try to see if the command
>> `docker run -p 8099:8099 -p 8098:8098 -p 8097:8097
>> apachebeam/flink1.9_job_server:latest` works?
>>
>> [1] https://forums.docker.com/t/should-docker-run-net-host-work/14215/28
>>
>> Best,
>> Jincheng
>> -------------
>> Twitter: https://twitter.com/sunjincheng121
>> -------------
>>
>>
>> Xander Song <ia...@gmail.com> 于2020年2月9日周日 上午11:52写道：
>>
>>> Do you have any suggestions for addressing this issue? I am unsure of
>>> what to try next.
>>>
>>> On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka <go...@google.com> wrote:
>>>
>>>> That seems to be a problem.
>>>>
>>>> When I try the command, I get
>>>>
>>>> $ telnet localhost 8099
>>>> Trying ::1...
>>>> Connected to localhost.
>>>> Escape character is '^]'.
>>>> �^CConnection closed by foreign host.
>>>>
>>>> On Fri, Feb 7, 2020 at 5:34 PM Xander Song <ia...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks for the response. After entering telnet localhost 8099, I
>>>>> receive
>>>>>
>>>>> Trying ::1...
>>>>>
>>>>> telnet: connect to address ::1: Connection refused
>>>>>
>>>>> Trying 127.0.0.1...
>>>>>
>>>>> telnet: connect to address 127.0.0.1: Connection refused
>>>>>
>>>>> telnet: Unable to connect to remote host
>>>>>
>>>>>
>>>>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka <go...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Seems that pipeline submission from sdk is not able to reach the job
>>>>>> server which was started in docker.
>>>>>>
>>>>>> Can you try running "telnet localhost 8099" to make sure that
>>>>>> pipeline submission can reach the job server.
>>>>>>
>>>>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song <ia...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I am having difficulty following the Python guide for running Beam
>>>>>>> on Flink <https://beam.apache.org/documentation/runners/flink/>. I
>>>>>>> created a virtual environment with Apache Beam installed, then I started up
>>>>>>> the JobService Docker container with
>>>>>>>
>>>>>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>>>>>
>>>>>>>
>>>>>>> I receive the following message confirming that the container is
>>>>>>> running.
>>>>>>>
>>>>>>>
>>>>>>> [main] INFO
>>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>>> ArtifactStagingService started on localhost:8098
>>>>>>>
>>>>>>> [main] INFO
>>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>>>>>>> ExpansionService started on localhost:8097
>>>>>>>
>>>>>>> [main] INFO
>>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>>> JobService started on localhost:8099
>>>>>>>
>>>>>>>
>>>>>>> In another terminal, I execute a Beam script called
>>>>>>> test_beam_local_flink.py based on the example.
>>>>>>>
>>>>>>>
>>>>>>> from __future__ import print_function
>>>>>>> import apache_beamfrom apache_beam.options.pipeline_options import PipelineOptions
>>>>>>>
>>>>>>> data = [1,2,3]
>>>>>>>
>>>>>>> options = PipelineOptions([
>>>>>>> "--runner=PortableRunner",
>>>>>>> "--job_endpoint=localhost:8099",
>>>>>>> "--environment_type=LOOPBACK"])
>>>>>>> with apache_beam.Pipeline(options=options) as p:
>>>>>>> video_collection = (
>>>>>>> p | apache_beam.Create(data)
>>>>>>> | apache_beam.Map(lambda x: x + 1)
>>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>>> )
>>>>>>> print('Done')
>>>>>>>
>>>>>>> After a wait, I get the following traceback.
>>>>>>>
>>>>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only.
>>>>>>>
>>>>>>> 'You are using Apache Beam with Python 2. '
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>>
>>>>>>> File "test_beam_local_flink.py", line 18, in <module>
>>>>>>>
>>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 481, in __exit__
>>>>>>>
>>>>>>> self.run().wait_until_finish()
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 461, in run
>>>>>>>
>>>>>>> self._options).run(False)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 474, in run
>>>>>>>
>>>>>>> return self.runner.run_pipeline(self, self._options)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 220, in run_pipeline
>>>>>>>
>>>>>>> job_service = self.create_job_service(options)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 136, in create_job_service
>>>>>>>
>>>>>>> return server.start()
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py", line 59, in start
>>>>>>>
>>>>>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 140, in result
>>>>>>>
>>>>>>> self._block(timeout)
>>>>>>>
>>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 86, in _block
>>>>>>>
>>>>>>> raise grpc.FutureTimeoutError()
>>>>>>>
>>>>>>> grpc.FutureTimeoutError
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any help is greatly appreciated.
>>>>>>>
>>>>>>>
Re: Running Beam on Flink
Posted by Xander Song <ia...@gmail.com>.
Hi Jincheng,
Thanks for your help. Yes, I am using Mac. Your suggestion allowed me to
submit the job on port 8099. However, I am now encountering a different
error message.
ERROR:root:java.net.ConnectException: Connection refused
Traceback (most recent call last):
File "test_beam_local_flink.py", line 18, in <module>
| apache_beam.Map(lambda x: print(x))
File
"/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py",
line 481, in __exit__
self.run().wait_until_finish()
File
"/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py",
line 461, in run
self._options).run(False)
File
"/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py",
line 474, in run
return self.runner.run_pipeline(self, self._options)
File
"/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 334, in run_pipeline
result.wait_until_finish()
File
"/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 455, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline
BeamApp-root-0209182021-fc6ec677_c351e1bb-d428-4475-9910-4544d8ec1de4
failed in state FAILED: java.net.ConnectException: Connection refused
I've attached a text file containing the output from the terminal running
the Docker container in case that is informative (it is quite lengthy). Any
suggestions are appreciated.
Best,
Xander
On Sun, Feb 9, 2020 at 2:44 AM jincheng sun <su...@gmail.com>
wrote:
>
> Hi Xander,
>
> Are you using Mac? The option --net=host doesn't work on Mac[1]. Could you
> try to see if the command
> `docker run -p 8099:8099 -p 8098:8098 -p 8097:8097
> apachebeam/flink1.9_job_server:latest` works?
>
> [1] https://forums.docker.com/t/should-docker-run-net-host-work/14215/28
>
> Best,
> Jincheng
> -------------
> Twitter: https://twitter.com/sunjincheng121
> -------------
>
>
> Xander Song <ia...@gmail.com> 于2020年2月9日周日 上午11:52写道：
>
>> Do you have any suggestions for addressing this issue? I am unsure of
>> what to try next.
>>
>> On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka <go...@google.com> wrote:
>>
>>> That seems to be a problem.
>>>
>>> When I try the command, I get
>>>
>>> $ telnet localhost 8099
>>> Trying ::1...
>>> Connected to localhost.
>>> Escape character is '^]'.
>>> �^CConnection closed by foreign host.
>>>
>>> On Fri, Feb 7, 2020 at 5:34 PM Xander Song <ia...@gmail.com>
>>> wrote:
>>>
>>>> Thanks for the response. After entering telnet localhost 8099, I
>>>> receive
>>>>
>>>> Trying ::1...
>>>>
>>>> telnet: connect to address ::1: Connection refused
>>>>
>>>> Trying 127.0.0.1...
>>>>
>>>> telnet: connect to address 127.0.0.1: Connection refused
>>>>
>>>> telnet: Unable to connect to remote host
>>>>
>>>>
>>>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka <go...@google.com> wrote:
>>>>
>>>>> Seems that pipeline submission from sdk is not able to reach the job
>>>>> server which was started in docker.
>>>>>
>>>>> Can you try running "telnet localhost 8099" to make sure that pipeline
>>>>> submission can reach the job server.
>>>>>
>>>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song <ia...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am having difficulty following the Python guide for running Beam
>>>>>> on Flink <https://beam.apache.org/documentation/runners/flink/>. I
>>>>>> created a virtual environment with Apache Beam installed, then I started up
>>>>>> the JobService Docker container with
>>>>>>
>>>>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>>>>
>>>>>>
>>>>>> I receive the following message confirming that the container is
>>>>>> running.
>>>>>>
>>>>>>
>>>>>> [main] INFO
>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>> ArtifactStagingService started on localhost:8098
>>>>>>
>>>>>> [main] INFO
>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>>>>>> ExpansionService started on localhost:8097
>>>>>>
>>>>>> [main] INFO
>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>> JobService started on localhost:8099
>>>>>>
>>>>>>
>>>>>> In another terminal, I execute a Beam script called
>>>>>> test_beam_local_flink.py based on the example.
>>>>>>
>>>>>>
>>>>>> from __future__ import print_function
>>>>>> import apache_beamfrom apache_beam.options.pipeline_options import PipelineOptions
>>>>>>
>>>>>> data = [1,2,3]
>>>>>>
>>>>>> options = PipelineOptions([
>>>>>> "--runner=PortableRunner",
>>>>>> "--job_endpoint=localhost:8099",
>>>>>> "--environment_type=LOOPBACK"])
>>>>>> with apache_beam.Pipeline(options=options) as p:
>>>>>> video_collection = (
>>>>>> p | apache_beam.Create(data)
>>>>>> | apache_beam.Map(lambda x: x + 1)
>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>> )
>>>>>> print('Done')
>>>>>>
>>>>>> After a wait, I get the following traceback.
>>>>>>
>>>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only.
>>>>>>
>>>>>> 'You are using Apache Beam with Python 2. '
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>
>>>>>> File "test_beam_local_flink.py", line 18, in <module>
>>>>>>
>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>>
>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 481, in __exit__
>>>>>>
>>>>>> self.run().wait_until_finish()
>>>>>>
>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 461, in run
>>>>>>
>>>>>> self._options).run(False)
>>>>>>
>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 474, in run
>>>>>>
>>>>>> return self.runner.run_pipeline(self, self._options)
>>>>>>
>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 220, in run_pipeline
>>>>>>
>>>>>> job_service = self.create_job_service(options)
>>>>>>
>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 136, in create_job_service
>>>>>>
>>>>>> return server.start()
>>>>>>
>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py", line 59, in start
>>>>>>
>>>>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>>>>
>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 140, in result
>>>>>>
>>>>>> self._block(timeout)
>>>>>>
>>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 86, in _block
>>>>>>
>>>>>> raise grpc.FutureTimeoutError()
>>>>>>
>>>>>> grpc.FutureTimeoutError
>>>>>>
>>>>>>
>>>>>>
>>>>>> Any help is greatly appreciated.
>>>>>>
>>>>>>
Re: Running Beam on Flink
Posted by jincheng sun <su...@gmail.com>.
Hi Xander,
Are you using Mac? The option --net=host doesn't work on Mac[1]. Could you
try to see if the command
`docker run -p 8099:8099 -p 8098:8098 -p 8097:8097
apachebeam/flink1.9_job_server:latest` works?
[1] https://forums.docker.com/t/should-docker-run-net-host-work/14215/28
Best,
Jincheng
-------------
Twitter: https://twitter.com/sunjincheng121
-------------
Xander Song <ia...@gmail.com> 于2020年2月9日周日 上午11:52写道：
> Do you have any suggestions for addressing this issue? I am unsure of what
> to try next.
>
> On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka <go...@google.com> wrote:
>
>> That seems to be a problem.
>>
>> When I try the command, I get
>>
>> $ telnet localhost 8099
>> Trying ::1...
>> Connected to localhost.
>> Escape character is '^]'.
>> �^CConnection closed by foreign host.
>>
>> On Fri, Feb 7, 2020 at 5:34 PM Xander Song <ia...@gmail.com> wrote:
>>
>>> Thanks for the response. After entering telnet localhost 8099, I receive
>>>
>>> Trying ::1...
>>>
>>> telnet: connect to address ::1: Connection refused
>>>
>>> Trying 127.0.0.1...
>>>
>>> telnet: connect to address 127.0.0.1: Connection refused
>>>
>>> telnet: Unable to connect to remote host
>>>
>>>
>>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka <go...@google.com> wrote:
>>>
>>>> Seems that pipeline submission from sdk is not able to reach the job
>>>> server which was started in docker.
>>>>
>>>> Can you try running "telnet localhost 8099" to make sure that pipeline
>>>> submission can reach the job server.
>>>>
>>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song <ia...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am having difficulty following the Python guide for running Beam on
>>>>> Flink <https://beam.apache.org/documentation/runners/flink/>. I
>>>>> created a virtual environment with Apache Beam installed, then I started up
>>>>> the JobService Docker container with
>>>>>
>>>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>>>
>>>>>
>>>>> I receive the following message confirming that the container is
>>>>> running.
>>>>>
>>>>>
>>>>> [main] INFO
>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>> ArtifactStagingService started on localhost:8098
>>>>>
>>>>> [main] INFO
>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>>>>> ExpansionService started on localhost:8097
>>>>>
>>>>> [main] INFO
>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>> JobService started on localhost:8099
>>>>>
>>>>>
>>>>> In another terminal, I execute a Beam script called
>>>>> test_beam_local_flink.py based on the example.
>>>>>
>>>>>
>>>>> from __future__ import print_function
>>>>> import apache_beamfrom apache_beam.options.pipeline_options import PipelineOptions
>>>>>
>>>>> data = [1,2,3]
>>>>>
>>>>> options = PipelineOptions([
>>>>> "--runner=PortableRunner",
>>>>> "--job_endpoint=localhost:8099",
>>>>> "--environment_type=LOOPBACK"])
>>>>> with apache_beam.Pipeline(options=options) as p:
>>>>> video_collection = (
>>>>> p | apache_beam.Create(data)
>>>>> | apache_beam.Map(lambda x: x + 1)
>>>>> | apache_beam.Map(lambda x: print(x))
>>>>> )
>>>>> print('Done')
>>>>>
>>>>> After a wait, I get the following traceback.
>>>>>
>>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only.
>>>>>
>>>>> 'You are using Apache Beam with Python 2. '
>>>>>
>>>>> Traceback (most recent call last):
>>>>>
>>>>> File "test_beam_local_flink.py", line 18, in <module>
>>>>>
>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>
>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 481, in __exit__
>>>>>
>>>>> self.run().wait_until_finish()
>>>>>
>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 461, in run
>>>>>
>>>>> self._options).run(False)
>>>>>
>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 474, in run
>>>>>
>>>>> return self.runner.run_pipeline(self, self._options)
>>>>>
>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 220, in run_pipeline
>>>>>
>>>>> job_service = self.create_job_service(options)
>>>>>
>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 136, in create_job_service
>>>>>
>>>>> return server.start()
>>>>>
>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py", line 59, in start
>>>>>
>>>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>>>
>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 140, in result
>>>>>
>>>>> self._block(timeout)
>>>>>
>>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 86, in _block
>>>>>
>>>>> raise grpc.FutureTimeoutError()
>>>>>
>>>>> grpc.FutureTimeoutError
>>>>>
>>>>>
>>>>>
>>>>> Any help is greatly appreciated.
>>>>>
>>>>>
Re: Running Beam on Flink
Posted by Xander Song <ia...@gmail.com>.
Do you have any suggestions for addressing this issue? I am unsure of what
to try next.
On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka <go...@google.com> wrote:
> That seems to be a problem.
>
> When I try the command, I get
>
> $ telnet localhost 8099
> Trying ::1...
> Connected to localhost.
> Escape character is '^]'.
> �^CConnection closed by foreign host.
>
> On Fri, Feb 7, 2020 at 5:34 PM Xander Song <ia...@gmail.com> wrote:
>
>> Thanks for the response. After entering telnet localhost 8099, I receive
>>
>> Trying ::1...
>>
>> telnet: connect to address ::1: Connection refused
>>
>> Trying 127.0.0.1...
>>
>> telnet: connect to address 127.0.0.1: Connection refused
>>
>> telnet: Unable to connect to remote host
>>
>>
>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka <go...@google.com> wrote:
>>
>>> Seems that pipeline submission from sdk is not able to reach the job
>>> server which was started in docker.
>>>
>>> Can you try running "telnet localhost 8099" to make sure that pipeline
>>> submission can reach the job server.
>>>
>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song <ia...@gmail.com>
>>> wrote:
>>>
>>>> I am having difficulty following the Python guide for running Beam on
>>>> Flink <https://beam.apache.org/documentation/runners/flink/>. I
>>>> created a virtual environment with Apache Beam installed, then I started up
>>>> the JobService Docker container with
>>>>
>>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>>
>>>>
>>>> I receive the following message confirming that the container is
>>>> running.
>>>>
>>>>
>>>> [main] INFO
>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>> ArtifactStagingService started on localhost:8098
>>>>
>>>> [main] INFO
>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>>>> ExpansionService started on localhost:8097
>>>>
>>>> [main] INFO
>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>> JobService started on localhost:8099
>>>>
>>>>
>>>> In another terminal, I execute a Beam script called
>>>> test_beam_local_flink.py based on the example.
>>>>
>>>>
>>>> from __future__ import print_function
>>>> import apache_beamfrom apache_beam.options.pipeline_options import PipelineOptions
>>>>
>>>> data = [1,2,3]
>>>>
>>>> options = PipelineOptions([
>>>> "--runner=PortableRunner",
>>>> "--job_endpoint=localhost:8099",
>>>> "--environment_type=LOOPBACK"])
>>>> with apache_beam.Pipeline(options=options) as p:
>>>> video_collection = (
>>>> p | apache_beam.Create(data)
>>>> | apache_beam.Map(lambda x: x + 1)
>>>> | apache_beam.Map(lambda x: print(x))
>>>> )
>>>> print('Done')
>>>>
>>>> After a wait, I get the following traceback.
>>>>
>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only.
>>>>
>>>> 'You are using Apache Beam with Python 2. '
>>>>
>>>> Traceback (most recent call last):
>>>>
>>>> File "test_beam_local_flink.py", line 18, in <module>
>>>>
>>>> | apache_beam.Map(lambda x: print(x))
>>>>
>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 481, in __exit__
>>>>
>>>> self.run().wait_until_finish()
>>>>
>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 461, in run
>>>>
>>>> self._options).run(False)
>>>>
>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 474, in run
>>>>
>>>> return self.runner.run_pipeline(self, self._options)
>>>>
>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 220, in run_pipeline
>>>>
>>>> job_service = self.create_job_service(options)
>>>>
>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 136, in create_job_service
>>>>
>>>> return server.start()
>>>>
>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py", line 59, in start
>>>>
>>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>>
>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 140, in result
>>>>
>>>> self._block(timeout)
>>>>
>>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 86, in _block
>>>>
>>>> raise grpc.FutureTimeoutError()
>>>>
>>>> grpc.FutureTimeoutError
>>>>
>>>>
>>>>
>>>> Any help is greatly appreciated.
>>>>
>>>>
Re: Running Beam on Flink
Posted by Ankur Goenka <go...@google.com>.
That seems to be a problem.
When I try the command, I get
$ telnet localhost 8099
Trying ::1...
Connected to localhost.
Escape character is '^]'.
�^CConnection closed by foreign host.
On Fri, Feb 7, 2020 at 5:34 PM Xander Song <ia...@gmail.com> wrote:
> Thanks for the response. After entering telnet localhost 8099, I receive
>
> Trying ::1...
>
> telnet: connect to address ::1: Connection refused
>
> Trying 127.0.0.1...
>
> telnet: connect to address 127.0.0.1: Connection refused
>
> telnet: Unable to connect to remote host
>
>
> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka <go...@google.com> wrote:
>
>> Seems that pipeline submission from sdk is not able to reach the job
>> server which was started in docker.
>>
>> Can you try running "telnet localhost 8099" to make sure that pipeline
>> submission can reach the job server.
>>
>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song <ia...@gmail.com> wrote:
>>
>>> I am having difficulty following the Python guide for running Beam on
>>> Flink <https://beam.apache.org/documentation/runners/flink/>. I created
>>> a virtual environment with Apache Beam installed, then I started up the
>>> JobService Docker container with
>>>
>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>
>>>
>>> I receive the following message confirming that the container is running.
>>>
>>>
>>> [main] INFO
>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>> ArtifactStagingService started on localhost:8098
>>>
>>> [main] INFO
>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>>> ExpansionService started on localhost:8097
>>>
>>> [main] INFO
>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>> JobService started on localhost:8099
>>>
>>>
>>> In another terminal, I execute a Beam script called
>>> test_beam_local_flink.py based on the example.
>>>
>>>
>>> from __future__ import print_function
>>> import apache_beamfrom apache_beam.options.pipeline_options import PipelineOptions
>>>
>>> data = [1,2,3]
>>>
>>> options = PipelineOptions([
>>> "--runner=PortableRunner",
>>> "--job_endpoint=localhost:8099",
>>> "--environment_type=LOOPBACK"])
>>> with apache_beam.Pipeline(options=options) as p:
>>> video_collection = (
>>> p | apache_beam.Create(data)
>>> | apache_beam.Map(lambda x: x + 1)
>>> | apache_beam.Map(lambda x: print(x))
>>> )
>>> print('Done')
>>>
>>> After a wait, I get the following traceback.
>>>
>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only.
>>>
>>> 'You are using Apache Beam with Python 2. '
>>>
>>> Traceback (most recent call last):
>>>
>>> File "test_beam_local_flink.py", line 18, in <module>
>>>
>>> | apache_beam.Map(lambda x: print(x))
>>>
>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 481, in __exit__
>>>
>>> self.run().wait_until_finish()
>>>
>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 461, in run
>>>
>>> self._options).run(False)
>>>
>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 474, in run
>>>
>>> return self.runner.run_pipeline(self, self._options)
>>>
>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 220, in run_pipeline
>>>
>>> job_service = self.create_job_service(options)
>>>
>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 136, in create_job_service
>>>
>>> return server.start()
>>>
>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py", line 59, in start
>>>
>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>
>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 140, in result
>>>
>>> self._block(timeout)
>>>
>>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 86, in _block
>>>
>>> raise grpc.FutureTimeoutError()
>>>
>>> grpc.FutureTimeoutError
>>>
>>>
>>>
>>> Any help is greatly appreciated.
>>>
>>>
Re: Running Beam on Flink
Posted by Xander Song <ia...@gmail.com>.
Thanks for the response. After entering telnet localhost 8099, I receive
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host
On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka <go...@google.com> wrote:
> Seems that pipeline submission from sdk is not able to reach the job
> server which was started in docker.
>
> Can you try running "telnet localhost 8099" to make sure that pipeline
> submission can reach the job server.
>
> On Thu, Feb 6, 2020 at 8:16 PM Xander Song <ia...@gmail.com> wrote:
>
>> I am having difficulty following the Python guide for running Beam on
>> Flink <https://beam.apache.org/documentation/runners/flink/>. I created
>> a virtual environment with Apache Beam installed, then I started up the
>> JobService Docker container with
>>
>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>
>>
>> I receive the following message confirming that the container is running.
>>
>>
>> [main] INFO
>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>> ArtifactStagingService started on localhost:8098
>>
>> [main] INFO
>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>> ExpansionService started on localhost:8097
>>
>> [main] INFO
>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>> JobService started on localhost:8099
>>
>>
>> In another terminal, I execute a Beam script called
>> test_beam_local_flink.py based on the example.
>>
>>
>> from __future__ import print_function
>> import apache_beamfrom apache_beam.options.pipeline_options import PipelineOptions
>>
>> data = [1,2,3]
>>
>> options = PipelineOptions([
>> "--runner=PortableRunner",
>> "--job_endpoint=localhost:8099",
>> "--environment_type=LOOPBACK"])
>> with apache_beam.Pipeline(options=options) as p:
>> video_collection = (
>> p | apache_beam.Create(data)
>> | apache_beam.Map(lambda x: x + 1)
>> | apache_beam.Map(lambda x: print(x))
>> )
>> print('Done')
>>
>> After a wait, I get the following traceback.
>>
>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only.
>>
>> 'You are using Apache Beam with Python 2. '
>>
>> Traceback (most recent call last):
>>
>> File "test_beam_local_flink.py", line 18, in <module>
>>
>> | apache_beam.Map(lambda x: print(x))
>>
>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 481, in __exit__
>>
>> self.run().wait_until_finish()
>>
>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 461, in run
>>
>> self._options).run(False)
>>
>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 474, in run
>>
>> return self.runner.run_pipeline(self, self._options)
>>
>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 220, in run_pipeline
>>
>> job_service = self.create_job_service(options)
>>
>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 136, in create_job_service
>>
>> return server.start()
>>
>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py", line 59, in start
>>
>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>
>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 140, in result
>>
>> self._block(timeout)
>>
>> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 86, in _block
>>
>> raise grpc.FutureTimeoutError()
>>
>> grpc.FutureTimeoutError
>>
>>
>>
>> Any help is greatly appreciated.
>>
>>
Re: Running Beam on Flink
Posted by Ankur Goenka <go...@google.com>.
Seems that pipeline submission from sdk is not able to reach the job server
which was started in docker.
Can you try running "telnet localhost 8099" to make sure that pipeline
submission can reach the job server.
On Thu, Feb 6, 2020 at 8:16 PM Xander Song <ia...@gmail.com> wrote:
> I am having difficulty following the Python guide for running Beam on
> Flink <https://beam.apache.org/documentation/runners/flink/>. I created a
> virtual environment with Apache Beam installed, then I started up the
> JobService Docker container with
>
> docker run --net=host apachebeam/flink1.9_job_server:latest
>
>
> I receive the following message confirming that the container is running.
>
>
> [main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> ArtifactStagingService started on localhost:8098
>
> [main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
> ExpansionService started on localhost:8097
>
> [main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> JobService started on localhost:8099
>
>
> In another terminal, I execute a Beam script called
> test_beam_local_flink.py based on the example.
>
>
> from __future__ import print_function
> import apache_beamfrom apache_beam.options.pipeline_options import PipelineOptions
>
> data = [1,2,3]
>
> options = PipelineOptions([
> "--runner=PortableRunner",
> "--job_endpoint=localhost:8099",
> "--environment_type=LOOPBACK"])
> with apache_beam.Pipeline(options=options) as p:
> video_collection = (
> p | apache_beam.Create(data)
> | apache_beam.Map(lambda x: x + 1)
> | apache_beam.Map(lambda x: print(x))
> )
> print('Done')
>
> After a wait, I get the following traceback.
>
> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only.
>
> 'You are using Apache Beam with Python 2. '
>
> Traceback (most recent call last):
>
> File "test_beam_local_flink.py", line 18, in <module>
>
> | apache_beam.Map(lambda x: print(x))
>
> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 481, in __exit__
>
> self.run().wait_until_finish()
>
> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 461, in run
>
> self._options).run(False)
>
> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", line 474, in run
>
> return self.runner.run_pipeline(self, self._options)
>
> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 220, in run_pipeline
>
> job_service = self.create_job_service(options)
>
> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 136, in create_job_service
>
> return server.start()
>
> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py", line 59, in start
>
> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>
> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 140, in result
>
> self._block(timeout)
>
> File "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", line 86, in _block
>
> raise grpc.FutureTimeoutError()
>
> grpc.FutureTimeoutError
>
>
>
> Any help is greatly appreciated.
>
>