You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Stephane Bonneaud <st...@fathomhealth.co> on 2018/05/23 21:28:49 UTC

Problem with the scheduler?

Hi,

I came about an issue with Airflow that I am wonder how to deal with it and thought I should reach out to see if you know about this issue and/or have advice on how to deal with it.

I have around 6 or 7 DAGs on Airflow that are scheduled to run at different intervals. Two of them should run every other day and, for some reason, they stopped running at some point, apparently not at the same time. So, they were running properly every other day, but then, they just stopped being scheduled, possibly around the same time, though I suspect that one was not scheduled anymore before the other one. Also, it does not seem to happen with my other DAGs, which are scheduled to run mostly daily, or bi-monthly. So maybe it has something to do with my scheduling string?

Here is the scheduling string for each of the DAGs: '0 7 */2 * *’ and '0 7 2/2 * *’
This seems to be the proper CRON string.

Have you seen that problem before?
How would you go about debugging this or do you have any advice/pointers for me?

Thank you very much for your time and help,
Have a wonderful rest of the day,

Best regards,
Stéphane

http://stephanebonneaud.com
(401) 580-0817

Re: Problem with the scheduler?

Posted by Maxime Beauchemin <ma...@gmail.com>.
Also note that for example when setting up monthly jobs, the job with an
execution_date of  `2018-02-01` will be triggered soon after the wall clock
hits `2018-03-01`, and that your start_date for the tasks in the DAG need
to be prior to that execution_date, not the time at which you're expecting
the job to trigger.

Max

On Thu, May 24, 2018 at 9:23 AM Victor Noagbodji <
vnoagbodji@amplify-nation.com> wrote:

> Hi Stephane,
>
> First cron string does not look correct. It says the 2nd of each month.
> Check here: https://crontab.guru/#0_7_/2_*_*
>
> However second looks okay to me. It says 2(-31)/2, every second day from 2
> till 31. Maybe the first string needs 1/2?
>
> Also, check this SO (sorry) answer:
> https://unix.stackexchange.com/questions/16093/how-can-i-tell-cron-to-run-a-command-every-other-day-odd-even/16094
>
> On May 23, 2018, at 5:28 PM, Stephane Bonneaud <stephane@fathomhealth.co
> <ma...@fathomhealth.co>> wrote:
>
> Hi,
>
> I came about an issue with Airflow that I am wonder how to deal with it
> and thought I should reach out to see if you know about this issue and/or
> have advice on how to deal with it.
>
> I have around 6 or 7 DAGs on Airflow that are scheduled to run at
> different intervals. Two of them should run every other day and, for some
> reason, they stopped running at some point, apparently not at the same
> time. So, they were running properly every other day, but then, they just
> stopped being scheduled, possibly around the same time, though I suspect
> that one was not scheduled anymore before the other one. Also, it does not
> seem to happen with my other DAGs, which are scheduled to run mostly daily,
> or bi-monthly. So maybe it has something to do with my scheduling string?
>
> Here is the scheduling string for each of the DAGs: '0 7 */2 * *’ and '0 7
> 2/2 * *’
> This seems to be the proper CRON string.
>
> Have you seen that problem before?
> How would you go about debugging this or do you have any advice/pointers
> for me?
>
> Thank you very much for your time and help,
> Have a wonderful rest of the day,
>
> Best regards,
> Stéphane
>
> http://stephanebonneaud.com
> (401) 580-0817
>
>

Re: Problem with the scheduler?

Posted by Victor Noagbodji <vn...@amplify-nation.com>.
Hi Stephane,

First cron string does not look correct. It says the 2nd of each month. Check here: https://crontab.guru/#0_7_/2_*_*

However second looks okay to me. It says 2(-31)/2, every second day from 2 till 31. Maybe the first string needs 1/2?

Also, check this SO (sorry) answer: https://unix.stackexchange.com/questions/16093/how-can-i-tell-cron-to-run-a-command-every-other-day-odd-even/16094

On May 23, 2018, at 5:28 PM, Stephane Bonneaud <st...@fathomhealth.co>> wrote:

Hi,

I came about an issue with Airflow that I am wonder how to deal with it and thought I should reach out to see if you know about this issue and/or have advice on how to deal with it.

I have around 6 or 7 DAGs on Airflow that are scheduled to run at different intervals. Two of them should run every other day and, for some reason, they stopped running at some point, apparently not at the same time. So, they were running properly every other day, but then, they just stopped being scheduled, possibly around the same time, though I suspect that one was not scheduled anymore before the other one. Also, it does not seem to happen with my other DAGs, which are scheduled to run mostly daily, or bi-monthly. So maybe it has something to do with my scheduling string?

Here is the scheduling string for each of the DAGs: '0 7 */2 * *’ and '0 7 2/2 * *’
This seems to be the proper CRON string.

Have you seen that problem before?
How would you go about debugging this or do you have any advice/pointers for me?

Thank you very much for your time and help,
Have a wonderful rest of the day,

Best regards,
Stéphane

http://stephanebonneaud.com
(401) 580-0817