You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by P Singh <bi...@gmail.com> on 2023/01/30 13:25:27 UTC
beam + flink + k8
Hi Team,
I am trying to run beam job on top of flink on my local machine
(kubernetes).
I have flink 1.14 and beam 2.43 images both running but when i submit the
job it's not reaching to the flink cluster and getting failed with below
error.
ERROR:apache_beam.utils.subprocess_server:Starting job service with
['java', '-jar',
'/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
'--flink-master', 'http://localhost:8081', '--artifacts-dir',
'/var/folders/n3/dqblsr792yj4kfs7xlfmdj540000gr/T/beam-tempvphhje07/artifacts6kjt60ch',
'--job-port', '57882', '--artifact-port', '0', '--expansion-port', '0']
ERROR:apache_beam.utils.subprocess_server:Error bringing up service
Traceback (most recent call last):
File
"/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
line 88, in start
raise RuntimeError(
RuntimeError: Service failed to start up with error 1
Any help would be appreciated.
Re: beam + flink + k8
Posted by Jan Lukavský <je...@seznam.cz>.
I'm not sure how exactly minikube exposes the jobmanager, but in GKE you
likely need to port-forward it, e.g.
$ kubectl port-forward svc/flink-jobmanager 8081:8081
This should make jobmanager accessible via localhost:8081. For
production cases you might want to use a different approach, like Flink
operator, etc.
Best,
Jan
On 2/1/23 17:08, P Singh wrote:
> Hi Jan,
>
> Thanks for the reply, I was able to submit the job to flink but it's
> failing due to an OOM issue so I am moving to the GKE. I got the flink
> UI there but submitted a job not appearing on flink UI. I am using the
> same script which I shared with you.. Do I need to make some changes
> for Google Kubernetes Environment?
>
> On Tue, 31 Jan 2023 at 20:20, Jan Lukavský <je...@seznam.cz> wrote:
>
> The script looks good to me, did you run the SDK harness? External
> environment needs the SDK harness to be run externally, see [1].
> Generally, the best option is DOCKER, but that usually does not
> work in k8s. For this, you might try PROCESS environment and build
> your own docker image for flink, which will contain the Beam
> harness, e.g. [2]. You will need to pass the environment config
> using --environment_config={"command": "/opt/apache/beam/boot"}.
>
> From the screenshot it seems, that the Flink UI is accessible, so
> this is the only option that comes to my mind. Did you check logs
> of the Flink jobmanager pod?
>
> Jan
>
> [1] https://beam.apache.org/documentation/runtime/sdk-harness-config/
>
> [2]
> https://github.com/PacktPublishing/Building-Big-Data-Pipelines-with-Apache-Beam/blob/main/env/docker/flink/Dockerfile
>
> On 1/31/23 13:33, P Singh wrote:
>> HI Jan,
>>
>> Thanks for your reply, please find attached script, I am newbie
>> with flink and minikube though i am trying to connect them by
>> script from local machine as suggested by flink kubernetes
>> documents link
>> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/>
>>
>> I have changed the log level to ERROR but didn't find much... Can
>> you please help me out how to run the script from inside the pod.
>>
>> On Tue, 31 Jan 2023 at 15:40, Jan Lukavský <je...@seznam.cz> wrote:
>>
>> Hi,
>>
>> can you please share the also the script itself? I'd say that
>> the problem is that the flink jobmanager is not accessible
>> through localhost:8081, because it runs inside the minikube.
>> You need to expose it outside of the minikube via [1], or run
>> the script from pod inside the minikube and access job
>> manager via flink-jobmanager:8081. I'm surprised that the log
>> didn't make this more obvious, though. Is it possible that
>> you changed the default log level to ERROR? Can you try DEBUG
>> or similar?
>>
>> Jan
>>
>> [1] https://minikube.sigs.k8s.io/docs/handbook/accessing/
>>
>> On 1/30/23 18:36, P Singh wrote:
>>> Hi Jan,
>>>
>>> Yeah I am using minikube and beam image with python 3.10.
>>>
>>> Please find the attached screenshots.
>>>
>>>
>>>
>>> On Mon, 30 Jan 2023 at 21:22, Jan Lukavský <je...@seznam.cz>
>>> wrote:
>>>
>>> Hi,
>>>
>>> can you please share the command-line and complete
>>> output of the script?
>>> Are you using minikube? Can you share list of your
>>> running pods?
>>>
>>> Jan
>>>
>>> On 1/30/23 14:25, P Singh wrote:
>>> > Hi Team,
>>> >
>>> > I am trying to run beam job on top of flink on my
>>> local machine
>>> > (kubernetes).
>>> >
>>> > I have flink 1.14 and beam 2.43 images both running
>>> but when i submit
>>> > the job it's not reaching to the flink cluster and
>>> getting failed with
>>> > below error.
>>> >
>>> > ERROR:apache_beam.utils.subprocess_server:Starting job
>>> service with
>>> > ['java', '-jar',
>>> >
>>> '/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
>>>
>>> > '--flink-master', 'http://localhost:8081',
>>> '--artifacts-dir',
>>> >
>>> '/var/folders/n3/dqblsr792yj4kfs7xlfmdj540000gr/T/beam-tempvphhje07/artifacts6kjt60ch',
>>>
>>> > '--job-port', '57882', '--artifact-port', '0',
>>> '--expansion-port', '0']
>>> > ERROR:apache_beam.utils.subprocess_server:Error
>>> bringing up service
>>> > Traceback (most recent call last):
>>> > File
>>> >
>>> "/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
>>>
>>> > line 88, in start
>>> > raise RuntimeError(
>>> > RuntimeError: Service failed to start up with error 1
>>> >
>>> > Any help would be appreciated.
>>>
Re: beam + flink + k8
Posted by P Singh <bi...@gmail.com>.
Hi Jan,
Thanks for the reply, I was able to submit the job to flink but it's
failing due to an OOM issue so I am moving to the GKE. I got the flink UI
there but submitted a job not appearing on flink UI. I am using the same
script which I shared with you.. Do I need to make some changes for Google
Kubernetes Environment?
On Tue, 31 Jan 2023 at 20:20, Jan Lukavský <je...@seznam.cz> wrote:
> The script looks good to me, did you run the SDK harness? External
> environment needs the SDK harness to be run externally, see [1]. Generally,
> the best option is DOCKER, but that usually does not work in k8s. For this,
> you might try PROCESS environment and build your own docker image for
> flink, which will contain the Beam harness, e.g. [2]. You will need to pass
> the environment config using --environment_config={"command":
> "/opt/apache/beam/boot"}.
>
> From the screenshot it seems, that the Flink UI is accessible, so this is
> the only option that comes to my mind. Did you check logs of the Flink
> jobmanager pod?
>
> Jan
>
> [1] https://beam.apache.org/documentation/runtime/sdk-harness-config/
>
> [2]
> https://github.com/PacktPublishing/Building-Big-Data-Pipelines-with-Apache-Beam/blob/main/env/docker/flink/Dockerfile
> On 1/31/23 13:33, P Singh wrote:
>
> HI Jan,
>
> Thanks for your reply, please find attached script, I am newbie with flink
> and minikube though i am trying to connect them by script from local
> machine as suggested by flink kubernetes documents link
> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/>
>
> I have changed the log level to ERROR but didn't find much... Can you
> please help me out how to run the script from inside the pod.
>
> On Tue, 31 Jan 2023 at 15:40, Jan Lukavský <je...@seznam.cz> wrote:
>
>> Hi,
>>
>> can you please share the also the script itself? I'd say that the problem
>> is that the flink jobmanager is not accessible through localhost:8081,
>> because it runs inside the minikube. You need to expose it outside of the
>> minikube via [1], or run the script from pod inside the minikube and access
>> job manager via flink-jobmanager:8081. I'm surprised that the log didn't
>> make this more obvious, though. Is it possible that you changed the default
>> log level to ERROR? Can you try DEBUG or similar?
>>
>> Jan
>>
>> [1] https://minikube.sigs.k8s.io/docs/handbook/accessing/
>> On 1/30/23 18:36, P Singh wrote:
>>
>> Hi Jan,
>>
>> Yeah I am using minikube and beam image with python 3.10.
>>
>> Please find the attached screenshots.
>>
>>
>>
>> On Mon, 30 Jan 2023 at 21:22, Jan Lukavský <je...@seznam.cz> wrote:
>>
>>> Hi,
>>>
>>> can you please share the command-line and complete output of the script?
>>> Are you using minikube? Can you share list of your running pods?
>>>
>>> Jan
>>>
>>> On 1/30/23 14:25, P Singh wrote:
>>> > Hi Team,
>>> >
>>> > I am trying to run beam job on top of flink on my local machine
>>> > (kubernetes).
>>> >
>>> > I have flink 1.14 and beam 2.43 images both running but when i submit
>>> > the job it's not reaching to the flink cluster and getting failed with
>>> > below error.
>>> >
>>> > ERROR:apache_beam.utils.subprocess_server:Starting job service with
>>> > ['java', '-jar',
>>> >
>>> '/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
>>>
>>> > '--flink-master', 'http://localhost:8081', '--artifacts-dir',
>>> >
>>> '/var/folders/n3/dqblsr792yj4kfs7xlfmdj540000gr/T/beam-tempvphhje07/artifacts6kjt60ch',
>>>
>>> > '--job-port', '57882', '--artifact-port', '0', '--expansion-port', '0']
>>> > ERROR:apache_beam.utils.subprocess_server:Error bringing up service
>>> > Traceback (most recent call last):
>>> > File
>>> >
>>> "/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
>>>
>>> > line 88, in start
>>> > raise RuntimeError(
>>> > RuntimeError: Service failed to start up with error 1
>>> >
>>> > Any help would be appreciated.
>>>
>>
Re: beam + flink + k8
Posted by Jan Lukavský <je...@seznam.cz>.
The script looks good to me, did you run the SDK harness? External
environment needs the SDK harness to be run externally, see [1].
Generally, the best option is DOCKER, but that usually does not work in
k8s. For this, you might try PROCESS environment and build your own
docker image for flink, which will contain the Beam harness, e.g. [2].
You will need to pass the environment config using
--environment_config={"command": "/opt/apache/beam/boot"}.
From the screenshot it seems, that the Flink UI is accessible, so this
is the only option that comes to my mind. Did you check logs of the
Flink jobmanager pod?
Jan
[1] https://beam.apache.org/documentation/runtime/sdk-harness-config/
[2]
https://github.com/PacktPublishing/Building-Big-Data-Pipelines-with-Apache-Beam/blob/main/env/docker/flink/Dockerfile
On 1/31/23 13:33, P Singh wrote:
> HI Jan,
>
> Thanks for your reply, please find attached script, I am newbie with
> flink and minikube though i am trying to connect them by script from
> local machine as suggested by flink kubernetes documents link
> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/>
>
> I have changed the log level to ERROR but didn't find much... Can you
> please help me out how to run the script from inside the pod.
>
> On Tue, 31 Jan 2023 at 15:40, Jan Lukavský <je...@seznam.cz> wrote:
>
> Hi,
>
> can you please share the also the script itself? I'd say that the
> problem is that the flink jobmanager is not accessible through
> localhost:8081, because it runs inside the minikube. You need to
> expose it outside of the minikube via [1], or run the script from
> pod inside the minikube and access job manager via
> flink-jobmanager:8081. I'm surprised that the log didn't make this
> more obvious, though. Is it possible that you changed the default
> log level to ERROR? Can you try DEBUG or similar?
>
> Jan
>
> [1] https://minikube.sigs.k8s.io/docs/handbook/accessing/
>
> On 1/30/23 18:36, P Singh wrote:
>> Hi Jan,
>>
>> Yeah I am using minikube and beam image with python 3.10.
>>
>> Please find the attached screenshots.
>>
>>
>>
>> On Mon, 30 Jan 2023 at 21:22, Jan Lukavský <je...@seznam.cz> wrote:
>>
>> Hi,
>>
>> can you please share the command-line and complete output of
>> the script?
>> Are you using minikube? Can you share list of your running pods?
>>
>> Jan
>>
>> On 1/30/23 14:25, P Singh wrote:
>> > Hi Team,
>> >
>> > I am trying to run beam job on top of flink on my local
>> machine
>> > (kubernetes).
>> >
>> > I have flink 1.14 and beam 2.43 images both running but
>> when i submit
>> > the job it's not reaching to the flink cluster and getting
>> failed with
>> > below error.
>> >
>> > ERROR:apache_beam.utils.subprocess_server:Starting job
>> service with
>> > ['java', '-jar',
>> >
>> '/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
>>
>> > '--flink-master', 'http://localhost:8081', '--artifacts-dir',
>> >
>> '/var/folders/n3/dqblsr792yj4kfs7xlfmdj540000gr/T/beam-tempvphhje07/artifacts6kjt60ch',
>>
>> > '--job-port', '57882', '--artifact-port', '0',
>> '--expansion-port', '0']
>> > ERROR:apache_beam.utils.subprocess_server:Error bringing up
>> service
>> > Traceback (most recent call last):
>> > File
>> >
>> "/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
>>
>> > line 88, in start
>> > raise RuntimeError(
>> > RuntimeError: Service failed to start up with error 1
>> >
>> > Any help would be appreciated.
>>
Re: beam + flink + k8
Posted by P Singh <bi...@gmail.com>.
HI Jan,
Thanks for your reply, please find attached script, I am newbie with flink
and minikube though i am trying to connect them by script from local
machine as suggested by flink kubernetes documents link
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/>
I have changed the log level to ERROR but didn't find much... Can you
please help me out how to run the script from inside the pod.
On Tue, 31 Jan 2023 at 15:40, Jan Lukavský <je...@seznam.cz> wrote:
> Hi,
>
> can you please share the also the script itself? I'd say that the problem
> is that the flink jobmanager is not accessible through localhost:8081,
> because it runs inside the minikube. You need to expose it outside of the
> minikube via [1], or run the script from pod inside the minikube and access
> job manager via flink-jobmanager:8081. I'm surprised that the log didn't
> make this more obvious, though. Is it possible that you changed the default
> log level to ERROR? Can you try DEBUG or similar?
>
> Jan
>
> [1] https://minikube.sigs.k8s.io/docs/handbook/accessing/
> On 1/30/23 18:36, P Singh wrote:
>
> Hi Jan,
>
> Yeah I am using minikube and beam image with python 3.10.
>
> Please find the attached screenshots.
>
>
>
> On Mon, 30 Jan 2023 at 21:22, Jan Lukavský <je...@seznam.cz> wrote:
>
>> Hi,
>>
>> can you please share the command-line and complete output of the script?
>> Are you using minikube? Can you share list of your running pods?
>>
>> Jan
>>
>> On 1/30/23 14:25, P Singh wrote:
>> > Hi Team,
>> >
>> > I am trying to run beam job on top of flink on my local machine
>> > (kubernetes).
>> >
>> > I have flink 1.14 and beam 2.43 images both running but when i submit
>> > the job it's not reaching to the flink cluster and getting failed with
>> > below error.
>> >
>> > ERROR:apache_beam.utils.subprocess_server:Starting job service with
>> > ['java', '-jar',
>> >
>> '/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
>>
>> > '--flink-master', 'http://localhost:8081', '--artifacts-dir',
>> >
>> '/var/folders/n3/dqblsr792yj4kfs7xlfmdj540000gr/T/beam-tempvphhje07/artifacts6kjt60ch',
>>
>> > '--job-port', '57882', '--artifact-port', '0', '--expansion-port', '0']
>> > ERROR:apache_beam.utils.subprocess_server:Error bringing up service
>> > Traceback (most recent call last):
>> > File
>> >
>> "/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
>>
>> > line 88, in start
>> > raise RuntimeError(
>> > RuntimeError: Service failed to start up with error 1
>> >
>> > Any help would be appreciated.
>>
>
Re: beam + flink + k8
Posted by Jan Lukavský <je...@seznam.cz>.
Hi,
can you please share the also the script itself? I'd say that the
problem is that the flink jobmanager is not accessible through
localhost:8081, because it runs inside the minikube. You need to expose
it outside of the minikube via [1], or run the script from pod inside
the minikube and access job manager via flink-jobmanager:8081. I'm
surprised that the log didn't make this more obvious, though. Is it
possible that you changed the default log level to ERROR? Can you try
DEBUG or similar?
Jan
[1] https://minikube.sigs.k8s.io/docs/handbook/accessing/
On 1/30/23 18:36, P Singh wrote:
> Hi Jan,
>
> Yeah I am using minikube and beam image with python 3.10.
>
> Please find the attached screenshots.
>
>
>
> On Mon, 30 Jan 2023 at 21:22, Jan Lukavský <je...@seznam.cz> wrote:
>
> Hi,
>
> can you please share the command-line and complete output of the
> script?
> Are you using minikube? Can you share list of your running pods?
>
> Jan
>
> On 1/30/23 14:25, P Singh wrote:
> > Hi Team,
> >
> > I am trying to run beam job on top of flink on my local machine
> > (kubernetes).
> >
> > I have flink 1.14 and beam 2.43 images both running but when i
> submit
> > the job it's not reaching to the flink cluster and getting
> failed with
> > below error.
> >
> > ERROR:apache_beam.utils.subprocess_server:Starting job service with
> > ['java', '-jar',
> >
> '/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
>
> > '--flink-master', 'http://localhost:8081', '--artifacts-dir',
> >
> '/var/folders/n3/dqblsr792yj4kfs7xlfmdj540000gr/T/beam-tempvphhje07/artifacts6kjt60ch',
>
> > '--job-port', '57882', '--artifact-port', '0',
> '--expansion-port', '0']
> > ERROR:apache_beam.utils.subprocess_server:Error bringing up service
> > Traceback (most recent call last):
> > File
> >
> "/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
>
> > line 88, in start
> > raise RuntimeError(
> > RuntimeError: Service failed to start up with error 1
> >
> > Any help would be appreciated.
>
Re: beam + flink + k8
Posted by P Singh <bi...@gmail.com>.
Hi Jan,
Yeah I am using minikube and beam image with python 3.10.
Please find the attached screenshots.
On Mon, 30 Jan 2023 at 21:22, Jan Lukavský <je...@seznam.cz> wrote:
> Hi,
>
> can you please share the command-line and complete output of the script?
> Are you using minikube? Can you share list of your running pods?
>
> Jan
>
> On 1/30/23 14:25, P Singh wrote:
> > Hi Team,
> >
> > I am trying to run beam job on top of flink on my local machine
> > (kubernetes).
> >
> > I have flink 1.14 and beam 2.43 images both running but when i submit
> > the job it's not reaching to the flink cluster and getting failed with
> > below error.
> >
> > ERROR:apache_beam.utils.subprocess_server:Starting job service with
> > ['java', '-jar',
> >
> '/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
>
> > '--flink-master', 'http://localhost:8081', '--artifacts-dir',
> >
> '/var/folders/n3/dqblsr792yj4kfs7xlfmdj540000gr/T/beam-tempvphhje07/artifacts6kjt60ch',
>
> > '--job-port', '57882', '--artifact-port', '0', '--expansion-port', '0']
> > ERROR:apache_beam.utils.subprocess_server:Error bringing up service
> > Traceback (most recent call last):
> > File
> >
> "/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
>
> > line 88, in start
> > raise RuntimeError(
> > RuntimeError: Service failed to start up with error 1
> >
> > Any help would be appreciated.
>
Re: beam + flink + k8
Posted by Jan Lukavský <je...@seznam.cz>.
Hi,
can you please share the command-line and complete output of the script?
Are you using minikube? Can you share list of your running pods?
Jan
On 1/30/23 14:25, P Singh wrote:
> Hi Team,
>
> I am trying to run beam job on top of flink on my local machine
> (kubernetes).
>
> I have flink 1.14 and beam 2.43 images both running but when i submit
> the job it's not reaching to the flink cluster and getting failed with
> below error.
>
> ERROR:apache_beam.utils.subprocess_server:Starting job service with
> ['java', '-jar',
> '/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
> '--flink-master', 'http://localhost:8081', '--artifacts-dir',
> '/var/folders/n3/dqblsr792yj4kfs7xlfmdj540000gr/T/beam-tempvphhje07/artifacts6kjt60ch',
> '--job-port', '57882', '--artifact-port', '0', '--expansion-port', '0']
> ERROR:apache_beam.utils.subprocess_server:Error bringing up service
> Traceback (most recent call last):
> File
> "/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
> line 88, in start
> raise RuntimeError(
> RuntimeError: Service failed to start up with error 1
>
> Any help would be appreciated.