[DISCUSS] Move all airflow infra hooks from settings.py to its own file

[Ping Zhang <pi...@umich.edu>]

Hi all,

Currently, those infra hooks
<https://github.com/apache/airflow/blob/8070a781b6292ce0795ceed6f36217dd61023dad/airflow/settings.py#L139-L234>
(e.g. task_policy, dag_policy etc) live in the settings.py, making it
bloated.

I am proposing to move those infra hooks to its own file. As for the file
name, how about `airflow_infra_hooks.py`? Any suggestions are welcome.

Please let me know your thoughts.

Thanks,

Ping

Re: [DISCUSS] Move all airflow infra hooks from settings.py to its own file

[Jarek Potiuk <ja...@potiuk.com>]

Yeah. I feel like having one place where you can add custom code is better,
especially that there are a number of conf settings that you can also add
your custom code (think hostname_callable) so having one "dedicated" module
at least as an entrypoint sounds reasonable and yeah - you can import from
other places if your code gets complex.

J.


On Sat, Aug 6, 2022 at 9:23 AM Ash Berlin-Taylor <as...@apache.org> wrote:

> I think for 99% of users this file is empty or tiny, so it works well as
> it is for most of our users.
>
> If you want to reorganize the file you can use python imports
>
> ```
> from mymodule import task_policy
> ```
>
> On 6 August 2022 01:13:14 BST, Ping Zhang <pi...@umich.edu> wrote:
>>
>> Hi all,
>>
>> Currently, those infra hooks
>> <https://github.com/apache/airflow/blob/8070a781b6292ce0795ceed6f36217dd61023dad/airflow/settings.py#L139-L234>
>> (e.g. task_policy, dag_policy etc) live in the settings.py, making it
>> bloated.
>>
>> I am proposing to move those infra hooks to its own file. As for the file
>> name, how about `airflow_infra_hooks.py`? Any suggestions are welcome.
>>
>> Please let me know your thoughts.
>>
>> Thanks,
>>
>> Ping
>>
>

Re: [DISCUSS] Move all airflow infra hooks from settings.py to its own file

[Ash Berlin-Taylor <as...@apache.org>]

I think for 99% of users this file is empty or tiny, so it works well as it is for most of our users.

If you want to reorganize the file you can use python imports

```
from mymodule import task_policy
```

On 6 August 2022 01:13:14 BST, Ping Zhang <pi...@umich.edu> wrote:
>Hi all,
>
>Currently, those infra hooks
><https://github.com/apache/airflow/blob/8070a781b6292ce0795ceed6f36217dd61023dad/airflow/settings.py#L139-L234>
>(e.g. task_policy, dag_policy etc) live in the settings.py, making it
>bloated.
>
>I am proposing to move those infra hooks to its own file. As for the file
>name, how about `airflow_infra_hooks.py`? Any suggestions are welcome.
>
>Please let me know your thoughts.
>
>Thanks,
>
>Ping