You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Robert Metzger <rm...@apache.org> on 2020/09/11 11:20:58 UTC

Re: How to schedule Flink Batch Job periodically or daily

Hi Sunitha,

(Note: You've emailed both the dev@ and user@ mailing list. Please only use
the user@ mailing list for questions on how to use Flink. I'm moving the
dev@ list to bcc)

Flink does not have facilities for scheduling batch jobs, and there are no
plans to add such a feature (this is not in the scope of Flink, there are
already a number of workflow management tools).

On Fri, Sep 11, 2020 at 1:10 PM s_penakalapati@yahoo.com <
s_penakalapati@yahoo.com> wrote:

> Hi Team,
>
> We have Flink Batch Jobs which needs to be scheduled as listed below:
> Case 1 :    2.00 UTC time  daily
> Case 2 :    Periodically 2 hours once
> Case 3:     Schedule based on an event
>
> Request you to help me on this,  How to approach all the 3 use cases.
> Can we use Oozie workflows or any better approach.
>
> Regards,
> Sunitha
>

Re: How to schedule Flink Batch Job periodically or daily

Posted by "s_penakalapati@yahoo.com" <s_...@yahoo.com>.

Hi Arvid,
Thanks a lot.
Will check Airflow and Cron-job options.
Regards,Sunitha.
On Monday, September 14, 2020, 05:23:43 PM GMT+5:30, Arvid Heise <ar...@ververica.com> wrote:

Hi Sunitha,
oozie is a valid approach, but I'd recommend to evaluate Airflow first [1]. It's much better maintained and easier to use.

Both tools are more used to compose complex workflows though. If you just need a repeated execution, I'd go with cron jobs. For example, you can completely rely on K8s and use cron jobs [2] there if you run on K8s anyways (which I'd recommend).

[1] https://airflow.apache.org/[2] https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
On Fri, Sep 11, 2020 at 1:21 PM Robert Metzger <rm...@apache.org> wrote:

Hi Sunitha,
(Note: You've emailed both the dev@ and user@ mailing list. Please only use the user@ mailing list for questions on how to use Flink. I'm moving the dev@ list to bcc)
Flink does not have facilities for scheduling batch jobs, and there are no plans to add such a feature (this is not in the scope of Flink, there are already a number of workflow management tools).

On Fri, Sep 11, 2020 at 1:10 PM s_penakalapati@yahoo.com <s_...@yahoo.com> wrote:

Hi Team,
We have Flink Batch Jobs which needs to be scheduled as listed below:Case 1 : 2.00 UTC time dailyCase 2 : Periodically 2 hours onceCase 3: Schedule based on an event
Request you to help me on this, How to approach all the 3 use cases. Can we use Oozie workflows or any better approach.
Regards,Sunitha

Arvid Heise | Senior Java Developer

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng

Re: How to schedule Flink Batch Job periodically or daily

Posted by Arvid Heise <ar...@ververica.com>.

Hi Sunitha,

oozie is a valid approach, but I'd recommend to evaluate Airflow first [1].
It's much better maintained and easier to use.

Both tools are more used to compose complex workflows though. If you just
need a repeated execution, I'd go with cron jobs. For example, you can
completely rely on K8s and use cron jobs [2] there if you run on K8s
anyways (which I'd recommend).

[1] https://airflow.apache.org/
[2] https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

On Fri, Sep 11, 2020 at 1:21 PM Robert Metzger <rm...@apache.org> wrote:

> Hi Sunitha,
>
> (Note: You've emailed both the dev@ and user@ mailing list. Please only
> use the user@ mailing list for questions on how to use Flink. I'm moving
> the dev@ list to bcc)
>
> Flink does not have facilities for scheduling batch jobs, and there are no
> plans to add such a feature (this is not in the scope of Flink, there are
> already a number of workflow management tools).
>
>
> On Fri, Sep 11, 2020 at 1:10 PM s_penakalapati@yahoo.com <
> s_penakalapati@yahoo.com> wrote:
>
>> Hi Team,
>>
>> We have Flink Batch Jobs which needs to be scheduled as listed below:
>> Case 1 :    2.00 UTC time  daily
>> Case 2 :    Periodically 2 hours once
>> Case 3:     Schedule based on an event
>>
>> Request you to help me on this,  How to approach all the 3 use cases.
>> Can we use Oozie workflows or any better approach.
>>
>> Regards,
>> Sunitha
>>
>

-- 

Arvid Heise | Senior Java Developer

<https://www.ververica.com/>

Follow us @VervericaData

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng