You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mattp- (via GitHub)" <gi...@apache.org> on 2025/08/15 22:58:04 UTC

[D] mp_start_method was removed via 11acefa462b6 [airflow]

GitHub user mattp- created a discussion: mp_start_method was removed via 11acefa462b6

### Apache Airflow version

main (development)

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

looks like 11acefa462b6 deleted code that provided functionality for mp_start_method, among other things.

### What you think should happen instead?

either remove mp_start_method from config, or re-add implementation.

### How to reproduce

 n/a

### Operating System

all

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)


GitHub link: https://github.com/apache/airflow/discussions/54564

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] mp_start_method was removed via 11acefa462b6 [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
GitHub user eladkal added a comment to the discussion: mp_start_method was removed via 11acefa462b6

cc @kaxil something we overlooked in the cleanup?

GitHub link: https://github.com/apache/airflow/discussions/54564#discussioncomment-14122158

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] mp_start_method was removed via 11acefa462b6 [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
GitHub user boring-cyborg[bot] added a comment to the discussion: mp_start_method was removed via 11acefa462b6

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.


GitHub link: https://github.com/apache/airflow/discussions/54564#discussioncomment-14122157

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] mp_start_method was removed via 11acefa462b6 [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: mp_start_method was removed via 11acefa462b6

Yeah. I have not seen it is in config still - feel free to remove it.

GitHub link: https://github.com/apache/airflow/discussions/54564#discussioncomment-14142288

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] mp_start_method was removed via 11acefa462b6 [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
GitHub user kaxil added a comment to the discussion: mp_start_method was removed via 11acefa462b6

Yeah, feel free to remove please.

GitHub link: https://github.com/apache/airflow/discussions/54564#discussioncomment-14142848

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] mp_start_method was removed via 11acefa462b6 [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
GitHub user kaxil added a comment to the discussion: mp_start_method was removed via 11acefa462b6

Yup, teething issues with Airflow 2 to 3 migration. PRs welcomed


GitHub link: https://github.com/apache/airflow/discussions/54564#discussioncomment-14122161

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] mp_start_method was removed via 11acefa462b6 [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: mp_start_method was removed via 11acefa462b6

> @kaxil sure, but mp_start_method no longer exists outside of the config. if someone wants to use a different multiprocessing method they no longer can / need to hack something. I discovered this looking at https://github.com/apache/airflow/pull/54510 , where I realized another annoyance, uvicorn also sets spawn in a way that will make it a default if it isn't already set, so there will be more unexpected behavior there if someone is expecting forking.

Yes. Annoyances when we removed things that were never intended to be used are expected. Following what @kaxil wrote - if you feel that your case is also bothering for others and that it is worth turning that into a featire - feel free to a) discuss in devlist, and b) - ideally at the same time - make a PR to implement it. Otherwise - bite the bullet and fight with the annoyance - possibly it;s only you.

I comvert it into a discussion, but if anyone wants to contribute some feature that is generic and useful enough - maybe resurrecting it from the past - opening PR is really best place to start.

GitHub link: https://github.com/apache/airflow/discussions/54564#discussioncomment-14122162

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] mp_start_method was removed via 11acefa462b6 [airflow]

Posted by "mattp- (via GitHub)" <gi...@apache.org>.
GitHub user mattp- added a comment to the discussion: mp_start_method was removed via 11acefa462b6

@kaxil sure, but mp_start_method no longer exists outside of the config. if someone wants to use a different multiprocessing method they no longer can / need to hack something. I discovered this looking at https://github.com/apache/airflow/pull/54510 , where I realized another annoyance, uvicorn also sets spawn in a way that will make it a default if it isn't already set, so there will be more unexpected behavior there if someone is expecting forking.

GitHub link: https://github.com/apache/airflow/discussions/54564#discussioncomment-14122160

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] mp_start_method was removed via 11acefa462b6 [airflow]

Posted by "mattp- (via GitHub)" <gi...@apache.org>.
GitHub user mattp- added a comment to the discussion: mp_start_method was removed via 11acefa462b6

@potiuk I don't necessarily have time to reimplement this at the moment (and don’t actually need it, just noticed and thought it was worth flagging),  but it’s most certainly a regression and bug that this was a configurable feature that doesn’t work at present. I’d suggest leaving it as an issue in case someone else stumbles on it in the meantime before it’s either fixed or removed. If removing it is acceptable I can do that PR pretty easily 🙂

GitHub link: https://github.com/apache/airflow/discussions/54564#discussioncomment-14122182

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] mp_start_method was removed via 11acefa462b6 [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
GitHub user kaxil added a comment to the discussion: mp_start_method was removed via 11acefa462b6

We swapped Multiprocessing with Task SDK machinery in https://github.com/apache/airflow/pull/44972

https://github.com/apache/airflow/commit/11acefa462b6b8fe2b43042e23ce3ccba7e878e3 was just a cleanup.

GitHub link: https://github.com/apache/airflow/discussions/54564#discussioncomment-14122159

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org