You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Yu Watanabe <yu...@gmail.com> on 2019/09/17 02:06:03 UTC
How to use google container registry for FlinkRunner ?
Hello.
For testing I would like to use image uploaded to google container registry.
How can I use images pulled from other than bintray.io ?
============================================================================
admin@ip-172-31-9-89:~$ gcloud container images list --repository
asia.gcr.io/[PROJECTNAME] <http://asia.gcr.io/%5BPROJECTNAME%5D>
NAME
asia.gcr.io/[PROJECTNAME]/beam <http://asia.gcr.io/%5BPROJECTNAME%5D/beam>
============================================================================
Looks like FlinkRunner (2.15.0) uses bintray repository as a default
behavior. As a result I am not able to use
Caused by: java.lang.Exception: The user defined 'open()' method caused an
exception: java.io.IOException: Received exit code 125 for command 'docker
run -d --network=host --env=DOCKER_MAC_CONTAINER=null --rm
ywatanabe-docker-apache.bintray.io/beam/python3:latest --id=1
--logging_endpoint=localhost:33787 --artifact_endpoint=localhost:41681
--provision_endpoint=localhost:45089 --control_endpoint=localhost:37045'.
stderr: Unable to find image '
ywatanabe-docker-apache.bintray.io/beam/python3:latest' locallydocker:
Error response from daemon: manifest for
ywatanabe-docker-apache.bintray.io/beam/python3:latest not found: manifest
unknown: The named manifest is not known to the registry.See 'docker run --
According to online doc, there is not parameter which controls which image
to use .
https://beam.apache.org/documentation/runners/flink/
Pipeline options I am using is below.
============================================================================
options = PipelineOptions([
"--runner=FlinkRunner",
"--flink_version=1.8",
"--flink_master_url=localhost:8081",
"--worker_harness_container_image=
asia.gcr.io/PROJECTNAME/beam/python3",
"--experiments=beam_fn_api"
])
============================================================================
Perhaps is there any environment variable to specify which image to use ?
Best Regards,
Yu Watanabe
--
Yu Watanabe
Weekend Freelancer who loves to challenge building data platform
yu.w.tennis@gmail.com
[image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image:
Twitter icon] <https://twitter.com/yuwtennis>
Re: How to use google container registry for FlinkRunner ?
Posted by Yu Watanabe <yu...@gmail.com>.
Kyle, Benjamin
Thank you for the quick reply.
And yes, adding "environment_config" to Pipeline options did the trick.
===========================================================
options = PipelineOptions([
"--runner=FlinkRunner",
"--flink_version=1.8",
"--flink_master_url=localhost:8081",
"--environment_config=
asia.gcr.io/PROJECTNAME/beam/python3",
"--experiments=beam_fn_api"
])
p = beam.Pipeline(options=options)
===========================================================
Thanks,
Yu Watanabe
On Tue, Sep 17, 2019 at 11:21 AM Benjamin Tan <be...@gmail.com>
wrote:
> Something like this:
>
> options = PipelineOptions(["--runner=PortableRunner",
> "--environment_config=apachebeam/python3.7_sdk
> ", # <---
> "--job_endpoint=dnnserver2:8099"])
>
> On 2019/09/17 02:14:00, Kyle Weaver <kc...@google.com> wrote:
> > I think environment_config is the option you are looking for.
> >
> > On Mon, Sep 16, 2019 at 7:06 PM Yu Watanabe <yu...@gmail.com>
> wrote:
> >
> > > Hello.
> > >
> > > For testing I would like to use image uploaded to google container
> > > registry.
> > > How can I use images pulled from other than bintray.io ?
> > >
> > >
> > >
> ============================================================================
> > > admin@ip-172-31-9-89:~$ gcloud container images list --repository
> > > asia.gcr.io/[PROJECTNAME] <http://asia.gcr.io/%5BPROJECTNAME%5D> <
> http://asia.gcr.io/%5BPROJECTNAME%5D>
> > > NAME
> > > asia.gcr.io/[PROJECTNAME]/beam
> <http://asia.gcr.io/%5BPROJECTNAME%5D/beam> <
> http://asia.gcr.io/%5BPROJECTNAME%5D/beam>
> > >
> > >
> ============================================================================
> > >
> > > Looks like FlinkRunner (2.15.0) uses bintray repository as a default
> > > behavior. As a result I am not able to use
> > >
> > > Caused by: java.lang.Exception: The user defined 'open()' method
> caused an
> > > exception: java.io.IOException: Received exit code 125 for command
> 'docker
> > > run -d --network=host --env=DOCKER_MAC_CONTAINER=null --rm
> > > ywatanabe-docker-apache.bintray.io/beam/python3:latest --id=1
> > > --logging_endpoint=localhost:33787 --artifact_endpoint=localhost:41681
> > > --provision_endpoint=localhost:45089
> --control_endpoint=localhost:37045'.
> > > stderr: Unable to find image '
> > > ywatanabe-docker-apache.bintray.io/beam/python3:latest' locallydocker:
> > > Error response from daemon: manifest for
> > > ywatanabe-docker-apache.bintray.io/beam/python3:latest not found:
> > > manifest unknown: The named manifest is not known to the registry.See
> > > 'docker run --
> > >
> > > According to online doc, there is not parameter which controls which
> image
> > > to use .
> > >
> > > https://beam.apache.org/documentation/runners/flink/
> > >
> > > Pipeline options I am using is below.
> > >
> > >
> > >
> ============================================================================
> > > options = PipelineOptions([
> > > "--runner=FlinkRunner",
> > > "--flink_version=1.8",
> > > "--flink_master_url=localhost:8081",
> > > "--worker_harness_container_image=
> > > asia.gcr.io/PROJECTNAME/beam/python3",
> > > "--experiments=beam_fn_api"
> > > ])
> > >
> > >
> ============================================================================
> > >
> > > Perhaps is there any environment variable to specify which image to
> use ?
> > >
> > > Best Regards,
> > > Yu Watanabe
> > >
> > > --
> > > Yu Watanabe
> > > Weekend Freelancer who loves to challenge building data platform
> > > yu.w.tennis@gmail.com
> > > [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1>
> [image:
> > > Twitter icon] <https://twitter.com/yuwtennis>
> > >
> > --
> > Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
> >
>
--
Yu Watanabe
Weekend Freelancer who loves to challenge building data platform
yu.w.tennis@gmail.com
[image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image:
Twitter icon] <https://twitter.com/yuwtennis>
Re: How to use google container registry for FlinkRunner ?
Posted by Benjamin Tan <be...@gmail.com>.
Something like this:
options = PipelineOptions(["--runner=PortableRunner",
"--environment_config=apachebeam/python3.7_sdk ", # <---
"--job_endpoint=dnnserver2:8099"])
On 2019/09/17 02:14:00, Kyle Weaver <kc...@google.com> wrote:
> I think environment_config is the option you are looking for.
>
> On Mon, Sep 16, 2019 at 7:06 PM Yu Watanabe <yu...@gmail.com> wrote:
>
> > Hello.
> >
> > For testing I would like to use image uploaded to google container
> > registry.
> > How can I use images pulled from other than bintray.io ?
> >
> >
> > ============================================================================
> > admin@ip-172-31-9-89:~$ gcloud container images list --repository
> > asia.gcr.io/[PROJECTNAME] <http://asia.gcr.io/%5BPROJECTNAME%5D>
> > NAME
> > asia.gcr.io/[PROJECTNAME]/beam <http://asia.gcr.io/%5BPROJECTNAME%5D/beam>
> >
> > ============================================================================
> >
> > Looks like FlinkRunner (2.15.0) uses bintray repository as a default
> > behavior. As a result I am not able to use
> >
> > Caused by: java.lang.Exception: The user defined 'open()' method caused an
> > exception: java.io.IOException: Received exit code 125 for command 'docker
> > run -d --network=host --env=DOCKER_MAC_CONTAINER=null --rm
> > ywatanabe-docker-apache.bintray.io/beam/python3:latest --id=1
> > --logging_endpoint=localhost:33787 --artifact_endpoint=localhost:41681
> > --provision_endpoint=localhost:45089 --control_endpoint=localhost:37045'.
> > stderr: Unable to find image '
> > ywatanabe-docker-apache.bintray.io/beam/python3:latest' locallydocker:
> > Error response from daemon: manifest for
> > ywatanabe-docker-apache.bintray.io/beam/python3:latest not found:
> > manifest unknown: The named manifest is not known to the registry.See
> > 'docker run --
> >
> > According to online doc, there is not parameter which controls which image
> > to use .
> >
> > https://beam.apache.org/documentation/runners/flink/
> >
> > Pipeline options I am using is below.
> >
> >
> > ============================================================================
> > options = PipelineOptions([
> > "--runner=FlinkRunner",
> > "--flink_version=1.8",
> > "--flink_master_url=localhost:8081",
> > "--worker_harness_container_image=
> > asia.gcr.io/PROJECTNAME/beam/python3",
> > "--experiments=beam_fn_api"
> > ])
> >
> > ============================================================================
> >
> > Perhaps is there any environment variable to specify which image to use ?
> >
> > Best Regards,
> > Yu Watanabe
> >
> > --
> > Yu Watanabe
> > Weekend Freelancer who loves to challenge building data platform
> > yu.w.tennis@gmail.com
> > [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image:
> > Twitter icon] <https://twitter.com/yuwtennis>
> >
> --
> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
>
Re: How to use google container registry for FlinkRunner ?
Posted by Kyle Weaver <kc...@google.com>.
I think environment_config is the option you are looking for.
On Mon, Sep 16, 2019 at 7:06 PM Yu Watanabe <yu...@gmail.com> wrote:
> Hello.
>
> For testing I would like to use image uploaded to google container
> registry.
> How can I use images pulled from other than bintray.io ?
>
>
> ============================================================================
> admin@ip-172-31-9-89:~$ gcloud container images list --repository
> asia.gcr.io/[PROJECTNAME] <http://asia.gcr.io/%5BPROJECTNAME%5D>
> NAME
> asia.gcr.io/[PROJECTNAME]/beam <http://asia.gcr.io/%5BPROJECTNAME%5D/beam>
>
> ============================================================================
>
> Looks like FlinkRunner (2.15.0) uses bintray repository as a default
> behavior. As a result I am not able to use
>
> Caused by: java.lang.Exception: The user defined 'open()' method caused an
> exception: java.io.IOException: Received exit code 125 for command 'docker
> run -d --network=host --env=DOCKER_MAC_CONTAINER=null --rm
> ywatanabe-docker-apache.bintray.io/beam/python3:latest --id=1
> --logging_endpoint=localhost:33787 --artifact_endpoint=localhost:41681
> --provision_endpoint=localhost:45089 --control_endpoint=localhost:37045'.
> stderr: Unable to find image '
> ywatanabe-docker-apache.bintray.io/beam/python3:latest' locallydocker:
> Error response from daemon: manifest for
> ywatanabe-docker-apache.bintray.io/beam/python3:latest not found:
> manifest unknown: The named manifest is not known to the registry.See
> 'docker run --
>
> According to online doc, there is not parameter which controls which image
> to use .
>
> https://beam.apache.org/documentation/runners/flink/
>
> Pipeline options I am using is below.
>
>
> ============================================================================
> options = PipelineOptions([
> "--runner=FlinkRunner",
> "--flink_version=1.8",
> "--flink_master_url=localhost:8081",
> "--worker_harness_container_image=
> asia.gcr.io/PROJECTNAME/beam/python3",
> "--experiments=beam_fn_api"
> ])
>
> ============================================================================
>
> Perhaps is there any environment variable to specify which image to use ?
>
> Best Regards,
> Yu Watanabe
>
> --
> Yu Watanabe
> Weekend Freelancer who loves to challenge building data platform
> yu.w.tennis@gmail.com
> [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image:
> Twitter icon] <https://twitter.com/yuwtennis>
>
--
Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com