You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Поротиков Станислав Вячеславович via user <us...@beam.apache.org> on 2023/12/04 06:53:03 UTC
Why do we need Job Server?
Hello!
I want to know which cases could lead me to use separate job server for submutting jobs to Flink Cluster?
Which cases we don't need it at all?
Best regards,
Stanislav Porotikov
RE: Why do we need Job Server?
Posted by Поротиков Станислав Вячеславович via user <us...@beam.apache.org>.
I guess it's about not documented --flink_submit_uber_jar pipeline option.
Best regards,
Stanislav Porotikov
From: Alexey Romanenko <ar...@gmail.com>
Sent: Tuesday, December 5, 2023 3:48 PM
To: user <us...@beam.apache.org>
Subject: Re: Why do we need Job Server?
Oh, interesting. I didn’t know about that possibility, thanks
—
Alexey
On 4 Dec 2023, at 18:14, Robert Bradshaw via user <us...@beam.apache.org>> wrote:
Note that this shouldn't be strictly necessary, e.g. for Python one can embed the pipeline definition into the jar itself which is then just uploaded as an "ordinary" flink executable jar to the Flink master: https://github.com/apache/beam/blob/release-2.52.0/sdks/python/apache_beam/runners/portability/abstract_job_service.py#L301 If Java doens't do this yet we should probably update it to do so.
On Mon, Dec 4, 2023 at 7:10 AM Alexey Romanenko <ar...@gmail.com>> wrote:
There are two modes to run a job with FlinkRunner - Portable and Classic. If you run a job server in Portable mode then you meed to start a JobService, configured with your Flink cluster, and submit your job through this. If you run a job in Classical mode (only Java SDK pipeline) then you don’t need it.
More information on this is here:
Apache Flink Runner<https://beam.apache.org/documentation/runners/flink/>
beam.apache.org<https://beam.apache.org/documentation/runners/flink/>
<favicon.ico><https://beam.apache.org/documentation/runners/flink/>
—
Alexey
On 4 Dec 2023, at 07:53, Поротиков Станислав Вячеславович via user <us...@beam.apache.org>> wrote:
Hello!
I want to know which cases could lead me to use separate job server for submutting jobs to Flink Cluster?
Which cases we don't need it at all?
Best regards,
Stanislav Porotikov
Re: Why do we need Job Server?
Posted by Alexey Romanenko <ar...@gmail.com>.
Oh, interesting. I didn’t know about that possibility, thanks
—
Alexey
> On 4 Dec 2023, at 18:14, Robert Bradshaw via user <us...@beam.apache.org> wrote:
>
> Note that this shouldn't be strictly necessary, e.g. for Python one can embed the pipeline definition into the jar itself which is then just uploaded as an "ordinary" flink executable jar to the Flink master: https://github.com/apache/beam/blob/release-2.52.0/sdks/python/apache_beam/runners/portability/abstract_job_service.py#L301 If Java doens't do this yet we should probably update it to do so.
>
> On Mon, Dec 4, 2023 at 7:10 AM Alexey Romanenko <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
>> There are two modes to run a job with FlinkRunner - Portable and Classic. If you run a job server in Portable mode then you meed to start a JobService, configured with your Flink cluster, and submit your job through this. If you run a job in Classical mode (only Java SDK pipeline) then you don’t need it.
>>
>> More information on this is here:
>> Apache Flink Runner
>> beam.apache.org
>> <favicon.ico>
>> <https://beam.apache.org/documentation/runners/flink/>Apache Flink Runner <https://beam.apache.org/documentation/runners/flink/>
>> beam.apache.org <https://beam.apache.org/documentation/runners/flink/> <favicon.ico> <https://beam.apache.org/documentation/runners/flink/>
>>
>> —
>> Alexey
>>
>>
>>> On 4 Dec 2023, at 07:53, Поротиков Станислав Вячеславович via user <user@beam.apache.org <ma...@beam.apache.org>> wrote:
>>>
>>> Hello!
>>> I want to know which cases could lead me to use separate job server for submutting jobs to Flink Cluster?
>>> Which cases we don't need it at all?
>>>
>>>
>>> Best regards,
>>> Stanislav Porotikov
>>
Re: Why do we need Job Server?
Posted by Robert Bradshaw via user <us...@beam.apache.org>.
Note that this shouldn't be strictly necessary, e.g. for Python one can
embed the pipeline definition into the jar itself which is then just
uploaded as an "ordinary" flink executable jar to the Flink master:
https://github.com/apache/beam/blob/release-2.52.0/sdks/python/apache_beam/runners/portability/abstract_job_service.py#L301
If Java doens't do this yet we should probably update it to do so.
On Mon, Dec 4, 2023 at 7:10 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> There are two modes to run a job with FlinkRunner - Portable and Classic.
> If you run a job server in Portable mode then you meed to start a
> JobService, configured with your Flink cluster, and submit your job through
> this. If you run a job in Classical mode (only Java SDK pipeline) then you
> don’t need it.
>
> More information on this is here:
> Apache Flink Runner <https://beam.apache.org/documentation/runners/flink/>
> beam.apache.org <https://beam.apache.org/documentation/runners/flink/>
> [image: favicon.ico]
> <https://beam.apache.org/documentation/runners/flink/>
> <https://beam.apache.org/documentation/runners/flink/>
>
> —
> Alexey
>
>
> On 4 Dec 2023, at 07:53, Поротиков Станислав Вячеславович via user <
> user@beam.apache.org> wrote:
>
> Hello!
> I want to know which cases could lead me to use separate job server for
> submutting jobs to Flink Cluster?
> Which cases we don't need it at all?
>
>
> Best regards,
> Stanislav Porotikov
>
>
>
Re: Why do we need Job Server?
Posted by Alexey Romanenko <ar...@gmail.com>.
There are two modes to run a job with FlinkRunner - Portable and Classic. If you run a job server in Portable mode then you meed to start a JobService, configured with your Flink cluster, and submit your job through this. If you run a job in Classical mode (only Java SDK pipeline) then you don’t need it.
More information on this is here:
https://beam.apache.org/documentation/runners/flink/
—
Alexey
> On 4 Dec 2023, at 07:53, Поротиков Станислав Вячеславович via user <us...@beam.apache.org> wrote:
>
> Hello!
> I want to know which cases could lead me to use separate job server for submutting jobs to Flink Cluster?
> Which cases we don't need it at all?
>
>
> Best regards,
> Stanislav Porotikov