how to submit jobs remotely when a rest proxy like nginx is used and REST endpoint is bind to loopback interface?

[Ming Li <br...@gmail.com>]

Hi,

The flink official document clearly states that "Simple mutual
authentication may be enabled by configuration if authentication of
connections to the REST endpoint is required, but we recommend to deploy a
“side car proxy”: Bind the REST endpoint to the loopback interface (or the
pod-local interface in Kubernetes) and start a REST proxy that
authenticates and forwards the requests to Flink. Examples for proxies that
Flink users have deployed are Envoy Proxy <https://www.envoyproxy.io/> or NGINX
with MOD_AUTH
<http://nginx.org/en/docs/http/ngx_http_auth_request_module.html>."



So I am wondering, in standalone mode when HA is not enabeld, when a rest
proxy like nginx is used, and rest endpoint is bind to the loopback
interface, how should we submit jobs remotely?



ps.

1. sample flink-conf.yaml settings, and nginx settings are as below showing:

rest.bind-port: 9091/rest.bind-address: 127.0.0.1 （this is the port and ip
where the rest endpoint bind itself to in the host where it is started）

rest.port: 9091/rest.address: 127.0.0.1 (this is the port and ip used by
rest clients when submit requests, so basically it should reach the above
rest.bind-port/rest.bind-address)

[image: image.png]

2. I know that we can use curl to request the nginx proxy, which
authenticates and forwards the request to flink, as below showing: curl -v
-u user1:user1 http://10.20.39.43:8080/config (which is the address where
nginx is listening to)

3. I know that  we can submit jobs from the host where job manager is
located, as below showing:

/opt/flink-1.12.2/bin/flink run -m 127.0.0.1:9091
/opt/flink-1.12.2/examples/batch/WordCount.jar --input /tmp/README.txt
--output /tmp/flink.test.txt11  ()

Thanks!
-- 
Best Regards
Michael Li

Re: how to submit jobs remotely when a rest proxy like nginx is used and REST endpoint is bind to loopback interface?

[Arvid Heise <ar...@apache.org>]

Hi Ming,

instead of using the command line interface to run Flink applications, you
should use the REST API [1].
You should first upload your jar in one call and then execute the job in
the second call.

The rest endpoint would be http://10.20.39.43:8080/
<http://10.20.39.43:8080/config>

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/rest_api.html#jars-jarid-run

On Fri, Apr 9, 2021 at 4:45 AM Ming Li <br...@gmail.com> wrote:

> Hi,
>
> The flink official document clearly states that "Simple mutual
> authentication may be enabled by configuration if authentication of
> connections to the REST endpoint is required, but we recommend to deploy a
> “side car proxy”: Bind the REST endpoint to the loopback interface (or the
> pod-local interface in Kubernetes) and start a REST proxy that
> authenticates and forwards the requests to Flink. Examples for proxies that
> Flink users have deployed are Envoy Proxy <https://www.envoyproxy.io/> or NGINX
> with MOD_AUTH
> <http://nginx.org/en/docs/http/ngx_http_auth_request_module.html>."
>
>
>
> So I am wondering, in standalone mode when HA is not enabeld, when a rest
> proxy like nginx is used, and rest endpoint is bind to the loopback
> interface, how should we submit jobs remotely?
>
>
>
> ps.
>
> 1. sample flink-conf.yaml settings, and nginx settings are as below
> showing:
>
> rest.bind-port: 9091/rest.bind-address: 127.0.0.1 （this is the port and ip
> where the rest endpoint bind itself to in the host where it is started）
>
> rest.port: 9091/rest.address: 127.0.0.1 (this is the port and ip used by
> rest clients when submit requests, so basically it should reach the above
> rest.bind-port/rest.bind-address)
>
> [image: image.png]
>
> 2. I know that we can use curl to request the nginx proxy, which
> authenticates and forwards the request to flink, as below showing: curl
> -v -u user1:user1 http://10.20.39.43:8080/config (which is the address
> where nginx is listening to)
>
> 3. I know that  we can submit jobs from the host where job manager is
> located, as below showing:
>
> /opt/flink-1.12.2/bin/flink run -m 127.0.0.1:9091
> /opt/flink-1.12.2/examples/batch/WordCount.jar --input /tmp/README.txt
> --output /tmp/flink.test.txt11  ()
>
> Thanks!
> --
> Best Regards
> Michael Li
>