REST job submission

[Flavio Pompermaier <po...@okkam.it>]

Hi to all,
in our ETL we need to call an external (REST) service once a job ends: we
extract informations about accumulators and we update the job status.
However this is only possible if using the CLI client: if we call the job
via the REST API o Web UI (that is very useful to decouple our UI from the
Flink cluster) then this is not possible, because the REST API cannot
execute any code after env.execute().
I think that this is a very huge limitation: first of all, when writing
(and debugging) a Flink job, you assume that you can call multiple times
execute() and use the returned JobExecutionResult.
In second instance, the binary client and the rest client behaves
differently (with the CLI client everything works as expected).

What do you think about this? Is this a bug or not?

PS: I think also that the REST client should not be aware of any jar or
class instance, it should just call the job manager with the proper class
name and jar id (plus other options of course).

Cheers,
Flavio

Re: REST job submission

[Flavio Pompermaier <po...@okkam.it>]

Done: https://issues.apache.org/jira/browse/FLINK-10879

On Wed, Nov 14, 2018 at 10:14 AM Chesnay Schepler <ch...@apache.org>
wrote:

> I wouldn't consider it a _bug_ in that sense, but agree that the current
> behavior isn't ideal. Running a job via the CLI or WebUI should behave
> the same way, as such please open a JIRA.
>
> On 12.11.2018 12:50, Flavio Pompermaier wrote:
> > Hi to all,
> > in our ETL we need to call an external (REST) service once a job ends: we
> > extract informations about accumulators and we update the job status.
> > However this is only possible if using the CLI client: if we call the job
> > via the REST API o Web UI (that is very useful to decouple our UI from
> the
> > Flink cluster) then this is not possible, because the REST API cannot
> > execute any code after env.execute().
> > I think that this is a very huge limitation: first of all, when writing
> > (and debugging) a Flink job, you assume that you can call multiple times
> > execute() and use the returned JobExecutionResult.
> > In second instance, the binary client and the rest client behaves
> > differently (with the CLI client everything works as expected).
> >
> > What do you think about this? Is this a bug or not?
> >
> > PS: I think also that the REST client should not be aware of any jar or
> > class instance, it should just call the job manager with the proper class
> > name and jar id (plus other options of course).
> >
> > Cheers,
> > Flavio
> >
>
>

Re: REST job submission

[Chesnay Schepler <ch...@apache.org>]

I wouldn't consider it a _bug_ in that sense, but agree that the current 
behavior isn't ideal. Running a job via the CLI or WebUI should behave 
the same way, as such please open a JIRA.

On 12.11.2018 12:50, Flavio Pompermaier wrote:
> Hi to all,
> in our ETL we need to call an external (REST) service once a job ends: we
> extract informations about accumulators and we update the job status.
> However this is only possible if using the CLI client: if we call the job
> via the REST API o Web UI (that is very useful to decouple our UI from the
> Flink cluster) then this is not possible, because the REST API cannot
> execute any code after env.execute().
> I think that this is a very huge limitation: first of all, when writing
> (and debugging) a Flink job, you assume that you can call multiple times
> execute() and use the returned JobExecutionResult.
> In second instance, the binary client and the rest client behaves
> differently (with the CLI client everything works as expected).
>
> What do you think about this? Is this a bug or not?
>
> PS: I think also that the REST client should not be aware of any jar or
> class instance, it should just call the job manager with the proper class
> name and jar id (plus other options of course).
>
> Cheers,
> Flavio
>