[GitHub] [airflow] chetratep opened a new issue #7852: airflow 1.10.9 can't stop backfilling and timezone issue

[GitBox <gi...@apache.org>]

chetratep opened a new issue #7852: airflow 1.10.9 can't stop backfilling and timezone issue 
URL: https://github.com/apache/airflow/issues/7852
 
 
   ▽
   ```
   from builtins import range
   from datetime import timedelta
   import pendulum
   import airflow
   from airflow.models import DAG
   from airflow.operators.bash_operator import BashOperator
   from airflow.operators.dummy_operator import DummyOperator
   
   args = {
       'owner': 'airflow',
       'start_date': airflow.utils.dates.days_ago(2),
       'depends_on_past': False,
   }
   
   dag = DAG(
       dag_id='800OOM_test',
       catchup=False,
       default_args=args,
       schedule_interval='0 7 * * *',
       dagrun_timeout=timedelta(minutes=60),
   )
   
   # [START task]
   oom_test = BashOperator(
       task_id='800OOM_test',
       bash_command='python /home/tmail/work/scheduler/testOOM.py > /dev/null 2>&1 ',
       dag=dag,
   )
   
   oom_test
   
   if __name__ == "__main__":
       dag.cli()
   ```
   
   I also have disable catchup in 
   
   > airflow.cfg
   ```
   $ cat airflow_home/airflow.cfg  | grep catchup
   # Turn off scheduler catchup by setting this to False.
   # will not do scheduler catchup if this is False,
   # DAG definition (catchup)
   catchup_by_default = False
   
   $ cat airflow_home/airflow.cfg  | grep timezone
   # Default timezone in case supplied date times are naive
   # can be utc (default), system, or any IANA timezone string (e.g. Europe/Amsterdam)
   default_timezone = Asia/Tokyo
   ```
   The dag execute immediately after turn the dag on UI.
   it could be the that the scheduler is not timezone aware or the catch up function is not working.
   
   So I also tested with pendulum and it has the same issue.
   
   ```
   from builtins import range
   from datetime import timedelta
   import pendulum
   import airflow
   from airflow.models import DAG
   from airflow.operators.bash_operator import BashOperator
   from airflow.operators.dummy_operator import DummyOperator
   
   args = {
       'owner': 'airflow',
       'start_date': pendulum.datetime(year=2020, month=3, day=23).astimezone('Asia/Tokyo'),
       'depends_on_past': False,
   }
   
   dag = DAG(
       dag_id='700OOM_test',
       catchup=False,
       default_args=args,
       schedule_interval='0 7 * * *',
       dagrun_timeout=timedelta(minutes=60),
   )
   
   # [START task]
   oom_test = BashOperator(
       task_id='700OOM_test',
       bash_command='python /home/tmail/work/scheduler/testOOM.py > /dev/null 2>&1 ',
       dag=dag,
   )
   
   oom_test
   
   if __name__ == "__main__":
       dag.cli()
   ```
   
   
   Environment is CentOS7. airflow is installed using pip command. 
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] chetratep commented on issue #7852: airflow 1.10.9 can't stop backfilling and timezone issue

[GitBox <gi...@apache.org>]

chetratep commented on issue #7852: airflow 1.10.9 can't stop backfilling and timezone issue 
URL: https://github.com/apache/airflow/issues/7852#issuecomment-603779967
 
 
   The **defalut_timezone** in **airflow.cfg** does not make the dag timezone aware until it we directly set the **start_date** with timezone. 
   I tried the following
   
   ```
   from builtins import range
   from datetime import timedelta
   import pendulum
   
   import airflow
   from airflow.models import DAG
   from airflow.operators.bash_operator import BashOperator
   from airflow.operators.dummy_operator import DummyOperator
   local_tz = pendulum.timezone("Asia/Tokyo")
   pnow = pendulum.now('Asia/Tokyo')
   
   args = {
       'owner': 'airflow',
       'start_date': pnow.subtract(hours=36),
       'depends_on_past': False,
   }
   
   dag = DAG(
       dag_id='OOM_test',
       catchup=False,
       default_args=args,
       schedule_interval='40 10 * * *',
       dagrun_timeout=timedelta(minutes=60),
   )
   
   # [START task]
   oom_test = BashOperator(
       task_id='oom_test',
       bash_command='python /home/tmail/work/scheduler/testOOM.py > /dev/null 2>&1 ',
       dag=dag,
   )
   print("Dag: {} TimeZone: {}".format(dag.dag_id, dag.timezone))
   oom_test
   
   ```
   **pnow.subtract(hours=36)** stops the task from executing immediately after enable it on UI.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services