program args size for running jobs

[Esteban Serrano <st...@gmail.com>]

Hi Flink users
Looking for some ideas on alternatives/workaround for running a job which
requires a large number of parameters (one of them being a long JSON
string). When all params are taken together, the resulting REST API URL
puts the size of the URL over the current 4096 bytes limit. Ideally, the
POST submit could allow named parameters in the body in addition to query
params, but was wondering if people have run into this limitation and have
found ways to work around it for submitting jobs.
The following JIRA [1] documents the current shortcoming for other
operations in the Web UI with the same sort of issue/limitation.

Best,
E

[1] - https://issues.apache.org/jira/browse/FLINK-8949

Re: program args size for running jobs

[Fabian Hueske <fh...@gmail.com>]

Hi Esteban,

If you need the parameters to configure specific operators (instead of the
over all flow), you could pass the parameters as a file using the
distributed cache [1].
Note, the docs point to the DataSet (batch) API, but the feature works the
same way for DataStream programs as well.

Best,
Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/batch/#distributed-cache

2018-05-22 23:57 GMT+02:00 Esteban Serrano <st...@gmail.com>:

> Hi Flink users
> Looking for some ideas on alternatives/workaround for running a job which
> requires a large number of parameters (one of them being a long JSON
> string). When all params are taken together, the resulting REST API URL
> puts the size of the URL over the current 4096 bytes limit. Ideally, the
> POST submit could allow named parameters in the body in addition to query
> params, but was wondering if people have run into this limitation and have
> found ways to work around it for submitting jobs.
> The following JIRA [1] documents the current shortcoming for other
> operations in the Web UI with the same sort of issue/limitation.
>
> Best,
> E
>
> [1] - https://issues.apache.org/jira/browse/FLINK-8949
>