You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jaketf (via GitHub)" <gi...@apache.org> on 2024/12/20 20:37:42 UTC
[PR] skip setproctitle in `task_runner` on Mac OS [airflow]
jaketf opened a new pull request, #45124:
URL: https://github.com/apache/airflow/pull/45124
On some newer versions of Mac OS setproctitle can cause segfault https://github.com/benoitc/gunicorn/issues/3021
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "jlaneve (via GitHub)" <gi...@apache.org>.
jlaneve commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1894474774
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
```suggestion
else:
from setproctitle import setproctitle
setproctitle("airflow scheduler -- DagFileProcessorManager")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "jlaneve (via GitHub)" <gi...@apache.org>.
jlaneve commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1894484958
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
this may actually be a GitHub bug, in the "Files changed" tab it shows the indentation being off, but in the conversation / timeline it shows the indentation as being correct (and my suggested change is unneeded indentation)
<img width="805" alt="Screenshot 2024-12-20 at 6 01 53 PM" src="https://github.com/user-attachments/assets/809973ab-a791-4cc4-a847-76ce93885cc2" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "jlaneve (via GitHub)" <gi...@apache.org>.
jlaneve commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1894474774
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
```suggestion
else:
from setproctitle import setproctitle
setproctitle("airflow scheduler -- DagFileProcessorManager")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "dvarrazzo (via GitHub)" <gi...@apache.org>.
dvarrazzo commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1896877203
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
Hello, I am the setproctitle author, although I have limited experience with macOS and no hardware to test with it.
@gershnik has been so kind to contribute expertise and bugfixes to the project in relation with macOS. Maybe he might be able to help?
I wonder if this problem relates to concurrency during initialisation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "gershnik (via GitHub)" <gi...@apache.org>.
gershnik commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1905810569
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@ashb Thank you, this is super helpful! (I am still unable to reproduce this even once - just tried it again after updating macOS)
With regards to threads, setproctitle doesn't by itself create any threads. Apple frameworks do so internally for their XPC with Launch Services but those sit dormant unless functionality that is using them is invoked. Also see below.
The crash happens in a child process post-fork on a main thread, early during setproctitle initialization in [CFBundleGetBundleWithIdentifier](https://developer.apple.com/documentation/corefoundation/1537139-cfbundlegetbundlewithidentifier?language=objc) call. It is called by setproctitle with a static argument[^1]:
```cpp
CFBundleGetBundleWithIdentifier(CFSTR("com.apple.LaunchServices"))
```
so it references no caller-supplied memory that can become invalid somehow. Thus, it is the _internal_ memory of CoreFoundation that is somehow corrupted at the time of this call. In other words the crash is "impossible" unless CoreFoundation itself is in a broken state.
Also note that any threads Apple might create hasn't been started yet - this call happens long before such functionality is invoked.
All of this, combined with the fact that the crash is very non-deterministic suggests that setproctitle is a victim here of something (potentially itself) using Apple APIs on another thread in parallel with fork.
So the question is whether this is what is going on. Are there any calls to to setproctitle (including importing it) or any other Apple-using library in the parent process that can happen in parallel with fork?
[^1]: `CFSTR` is an Apple macro to produce a statically allocated CFString constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1905608524
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@gershnik Native stack trace from Console.app:
```
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x1931c95d0 __pthread_kill + 8
1 libsystem_pthread.dylib 0x193201c20 pthread_kill + 288
2 libsystem_c.dylib 0x1930d81e0 raise + 32
3 libpython3.12.dylib 0x1068f0c68 faulthandler_fatal_error + 384
4 libsystem_platform.dylib 0x193232584 _sigtramp + 56
5 libsystem_trace.dylib 0x192f4f0a4 _os_log_preferences_refresh + 40
6 libsystem_trace.dylib 0x192f4fb20 os_log_type_enabled + 712
7 CoreFoundation 0x1932cbab8 _CFBundleCopyLoadedImagePathForPointer + 84
8 CoreFoundation 0x1933898b0 _CFBundleGetBundleWithIdentifier + 164
9 _setproctitle.cpython-312-darwin.so 0x104d82ffc darwin_set_process_title + 84
10 _setproctitle.cpython-312-darwin.so 0x104d838b8 init_ps_display + 208
11 _setproctitle.cpython-312-darwin.so 0x104d8359c spt_setup + 400
12 _setproctitle.cpython-312-darwin.so 0x104d83254 spt_getproctitle + 16
13 libpython3.12.dylib 0x1061f5170 cfunction_vectorcall_NOARGS.llvm.1866380503741956643 + 104
14 libpython3.12.dylib 0x105fe5dfc _PyEval_EvalFrameDefault + 156008
15 libpython3.12.dylib 0x10608b3a0 PyEval_EvalCode + 220
16 libpython3.12.dylib 0x1062148d4 builtin_exec + 396
17 libpython3.12.dylib 0x1061f50b4 cfunction_vectorcall_FASTCALL_KEYWORDS.llvm.1866380503741956643 + 92
18 libpython3.12.dylib 0x105fe9a10 _PyEval_EvalFrameDefault + 171388
19 libpython3.12.dylib 0x105f5c928 _PyObject_VectorcallTstate.llvm.2292412377633951376 + 84
20 libpython3.12.dylib 0x105fa3cbc object_vacall.llvm.2292412377633951376 + 240
21 libpython3.12.dylib 0x105fa3518 PyObject_CallMethodObjArgs + 108
22 libpython3.12.dylib 0x105f6353c PyImport_ImportModuleLevelObject + 3100
23 libpython3.12.dylib 0x105fdcf04 _PyEval_EvalFrameDefault + 119408
24 libpython3.12.dylib 0x1061dd854 method_vectorcall.llvm.12955693216709424543 + 296
25 libpython3.12.dylib 0x105fea118 _PyEval_EvalFrameDefault + 173188
26 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
27 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
28 libpython3.12.dylib 0x105fe615c _PyEval_EvalFrameDefault + 156872
29 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
30 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
31 libpython3.12.dylib 0x105fe9b48 _PyEval_EvalFrameDefault + 171700
32 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
33 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
34 libpython3.12.dylib 0x105fe615c _PyEval_EvalFrameDefault + 156872
35 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
36 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
37 libpython3.12.dylib 0x105fe615c _PyEval_EvalFrameDefault + 156872
38 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
39 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
40 libpython3.12.dylib 0x105fe615c _PyEval_EvalFrameDefault + 156872
41 libpython3.12.dylib 0x10608b3a0 PyEval_EvalCode + 220
42 libpython3.12.dylib 0x10608b1f4 run_mod.llvm.6674925059613253997 + 284
43 libpython3.12.dylib 0x106106730 pyrun_file + 156
44 libpython3.12.dylib 0x106105e70 _PyRun_SimpleFileObject + 268
45 libpython3.12.dylib 0x1061000fc _PyRun_AnyFileObject + 80
46 libpython3.12.dylib 0x1060fedd4 pymain_run_file_obj + 164
47 libpython3.12.dylib 0x1060fe438 pymain_run_file + 72
48 libpython3.12.dylib 0x1060fc774 Py_RunMain + 1124
49 libpython3.12.dylib 0x1060dcf7c pymain_main + 456
50 libpython3.12.dylib 0x1060dcda8 Py_BytesMain + 40
51 dyld 0x192e77154 start + 2476
```
and
```
-----------
Full Report
-----------
{"app_name":"python3.12","timestamp":"2025-01-07 15:12:30.00 +0000","app_version":"","slice_uuid":"4c4c44ea-5555-3144-a121-b75170b036a4","build_version":"","platform":1,"share_with_app_devs":0,"is_first_party":1,"bug_type":"309","os_version":"macOS 14.7 (23H124)","roots_installed":0,"incident_id":"38B0B6A5-AC4B-438D-9ED5-DDD274E676C1","name":"python3.12"}
{
"uptime" : 510000,
"procRole" : "Unspecified",
"version" : 2,
"userID" : 501,
"deployVersion" : 210,
"modelCode" : "Mac14,5",
"coalitionID" : 745,
"osVersion" : {
"train" : "macOS 14.7",
"build" : "23H124",
"releaseType" : "User"
},
"captureTime" : "2025-01-07 15:12:30.1013 +0000",
"codeSigningMonitor" : 1,
"incident" : "38B0B6A5-AC4B-438D-9ED5-DDD274E676C1",
"pid" : 51654,
"translated" : false,
"cpuType" : "ARM-64",
"roots_installed" : 0,
"bug_type" : "309",
"procLaunch" : "2025-01-07 15:12:30.0878 +0000",
"procStartAbsTime" : 12276109475883,
"procExitAbsTime" : 12276109794958,
"procName" : "python3.12",
"procPath" : "\/Users\/USER\/*\/python3.12",
"parentProc" : "python",
"parentPid" : 51646,
"coalitionName" : "com.github.wez.wezterm",
"crashReporterKey" : "5BB32C2E-55C1-DCA5-AE8B-3531EFE6FBEE",
"responsiblePid" : 804,
"responsibleProc" : "wezterm-gui",
"codeSigningID" : "-",
"codeSigningTeamID" : "",
"codeSigningFlags" : 570556961,
"codeSigningValidationCategory" : 10,
"codeSigningTrustLevel" : 4294967295,
"instructionByteStream" : {"beforePC":"fyMD1f17v6n9AwCRd+D\/l78DAJH9e8Go\/w9f1sADX9YQKYDSARAA1A==","atPC":"AwEAVH8jA9X9e7+p\/QMAkWzg\/5e\/AwCR\/XvBqP8PX9bAA1\/WcAqA0g=="},
"wakeTime" : 1751,
"sleepWakeUUID" : "92850A44-01E1-4036-91AC-9437DCBBCA46",
"sip" : "enabled",
"vmRegionInfo" : "0x104d60a8e is not in any region. Bytes after previous region: 2703 Bytes before following region: 128370\n REGION TYPE START - END [ VSIZE] PRT\/MAX SHRMOD REGION DETAIL\n MALLOC metadata 104d5c000-104d60000 [ 16K] rw-\/rwx SM=COW \n---> GAP OF 0x20000 BYTES\n __TEXT 104d80000-104d84000 [ 16K] r-x\/rwx SM=COW \/Users\/USER\/*\/_setproctitle.cpython-312-darwin.so",
"exception" : {"codes":"0x0000000000000001, 0x0000000104d60a8e","rawCodes":[1,4376103566],"type":"EXC_BAD_ACCESS","signal":"SIGSEGV","subtype":"KERN_INVALID_ADDRESS at 0x0000000104d60a8e"},
"termination" : {"flags":0,"code":11,"namespace":"SIGNAL","indicator":"Segmentation fault: 11","byProc":"python3.12","byPid":51654},
"vmregioninfo" : "0x104d60a8e is not in any region. Bytes after previous region: 2703 Bytes before following region: 128370\n REGION TYPE START - END [ VSIZE] PRT\/MAX SHRMOD REGION DETAIL\n MALLOC metadata 104d5c000-104d60000 [ 16K] rw-\/rwx SM=COW \n---> GAP OF 0x20000 BYTES\n __TEXT 104d80000-104d84000 [ 16K] r-x\/rwx SM=COW \/Users\/USER\/*\/_setproctitle.cpython-312-darwin.so",
"asi" : {"libsystem_c.dylib":["crashed on child side of fork pre-exec"]},
"extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
"faultingThread" : 0,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "gershnik (via GitHub)" <gi...@apache.org>.
gershnik commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1896913623
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@dvarrazzo I will definitely take a look after the holidays
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb merged PR #45124:
URL: https://github.com/apache/airflow/pull/45124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "dvarrazzo (via GitHub)" <gi...@apache.org>.
dvarrazzo commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1897036011
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@gershnik that's very appreciated. I wish you good holidays!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1900091124
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
I think this might be another incarnation of the threading issue . We should simply never do `thread+fork` in MacOS because this will eventually lead to SIGSEGV. So I am not sure it's anything setproctitle related, I think we should look where we are starting new threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #45124:
URL: https://github.com/apache/airflow/pull/45124#issuecomment-2676463643
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #45124: skip setproctitle in `task_runner` on Mac OS
URL: https://github.com/apache/airflow/pull/45124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "jlaneve (via GitHub)" <gi...@apache.org>.
jlaneve commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1894474774
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
```suggestion
else:
from setproctitle import setproctitle
setproctitle("airflow scheduler -- DagFileProcessorManager")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1894478809
##########
task_sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -214,9 +214,14 @@ def startup() -> tuple[RuntimeTaskInstance, Logger]:
msg = SUPERVISOR_COMMS.get_message()
if isinstance(msg, StartupDetails):
- from setproctitle import setproctitle
-
- setproctitle(f"airflow worker -- {msg.ti.id}")
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
Review Comment:
```suggestion
log.debug("Mac OS detected, skipping setproctitle")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1905608524
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@gershnik Native stack trace from Console.app:
```
VM Region Info: 0x104d60a8e is not in any region. Bytes after previous region: 2703 Bytes before following region: 128370
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
MALLOC metadata 104d5c000-104d60000 [ 16K] rw-/rwx SM=COW
---> GAP OF 0x20000 BYTES
__TEXT 104d80000-104d84000 [ 16K] r-x/rwx SM=COW /Users/USER/*/_setproctitle.cpython-312-darwin.so
Application Specific Information:
crashed on child side of fork pre-exec
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x1931c95d0 __pthread_kill + 8
1 libsystem_pthread.dylib 0x193201c20 pthread_kill + 288
2 libsystem_c.dylib 0x1930d81e0 raise + 32
3 libpython3.12.dylib 0x1068f0c68 faulthandler_fatal_error + 384
4 libsystem_platform.dylib 0x193232584 _sigtramp + 56
5 libsystem_trace.dylib 0x192f4f0a4 _os_log_preferences_refresh + 40
6 libsystem_trace.dylib 0x192f4fb20 os_log_type_enabled + 712
7 CoreFoundation 0x1932cbab8 _CFBundleCopyLoadedImagePathForPointer + 84
8 CoreFoundation 0x1933898b0 _CFBundleGetBundleWithIdentifier + 164
9 _setproctitle.cpython-312-darwin.so 0x104d82ffc darwin_set_process_title + 84
10 _setproctitle.cpython-312-darwin.so 0x104d838b8 init_ps_display + 208
11 _setproctitle.cpython-312-darwin.so 0x104d8359c spt_setup + 400
12 _setproctitle.cpython-312-darwin.so 0x104d83254 spt_getproctitle + 16
13 libpython3.12.dylib 0x1061f5170 cfunction_vectorcall_NOARGS.llvm.1866380503741956643 + 104
14 libpython3.12.dylib 0x105fe5dfc _PyEval_EvalFrameDefault + 156008
15 libpython3.12.dylib 0x10608b3a0 PyEval_EvalCode + 220
16 libpython3.12.dylib 0x1062148d4 builtin_exec + 396
17 libpython3.12.dylib 0x1061f50b4 cfunction_vectorcall_FASTCALL_KEYWORDS.llvm.1866380503741956643 + 92
18 libpython3.12.dylib 0x105fe9a10 _PyEval_EvalFrameDefault + 171388
19 libpython3.12.dylib 0x105f5c928 _PyObject_VectorcallTstate.llvm.2292412377633951376 + 84
20 libpython3.12.dylib 0x105fa3cbc object_vacall.llvm.2292412377633951376 + 240
21 libpython3.12.dylib 0x105fa3518 PyObject_CallMethodObjArgs + 108
22 libpython3.12.dylib 0x105f6353c PyImport_ImportModuleLevelObject + 3100
23 libpython3.12.dylib 0x105fdcf04 _PyEval_EvalFrameDefault + 119408
24 libpython3.12.dylib 0x1061dd854 method_vectorcall.llvm.12955693216709424543 + 296
25 libpython3.12.dylib 0x105fea118 _PyEval_EvalFrameDefault + 173188
26 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
27 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
28 libpython3.12.dylib 0x105fe615c _PyEval_EvalFrameDefault + 156872
29 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
30 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
31 libpython3.12.dylib 0x105fe9b48 _PyEval_EvalFrameDefault + 171700
32 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
33 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
34 libpython3.12.dylib 0x105fe615c _PyEval_EvalFrameDefault + 156872
35 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
36 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
37 libpython3.12.dylib 0x105fe615c _PyEval_EvalFrameDefault + 156872
38 libpython3.12.dylib 0x105ffb244 _PyObject_Call_Prepend + 296
39 libpython3.12.dylib 0x105ffac48 slot_tp_call + 116
40 libpython3.12.dylib 0x105fe615c _PyEval_EvalFrameDefault + 156872
41 libpython3.12.dylib 0x10608b3a0 PyEval_EvalCode + 220
42 libpython3.12.dylib 0x10608b1f4 run_mod.llvm.6674925059613253997 + 284
43 libpython3.12.dylib 0x106106730 pyrun_file + 156
44 libpython3.12.dylib 0x106105e70 _PyRun_SimpleFileObject + 268
45 libpython3.12.dylib 0x1061000fc _PyRun_AnyFileObject + 80
46 libpython3.12.dylib 0x1060fedd4 pymain_run_file_obj + 164
47 libpython3.12.dylib 0x1060fe438 pymain_run_file + 72
48 libpython3.12.dylib 0x1060fc774 Py_RunMain + 1124
49 libpython3.12.dylib 0x1060dcf7c pymain_main + 456
50 libpython3.12.dylib 0x1060dcda8 Py_BytesMain + 40
51 dyld 0x192e77154 start + 2476
```
and
```
-----------
Full Report
-----------
{"app_name":"python3.12","timestamp":"2025-01-07 15:12:30.00 +0000","app_version":"","slice_uuid":"4c4c44ea-5555-3144-a121-b75170b036a4","build_version":"","platform":1,"share_with_app_devs":0,"is_first_party":1,"bug_type":"309","os_version":"macOS 14.7 (23H124)","roots_installed":0,"incident_id":"38B0B6A5-AC4B-438D-9ED5-DDD274E676C1","name":"python3.12"}
{
"uptime" : 510000,
"procRole" : "Unspecified",
"version" : 2,
"userID" : 501,
"deployVersion" : 210,
"modelCode" : "Mac14,5",
"coalitionID" : 745,
"osVersion" : {
"train" : "macOS 14.7",
"build" : "23H124",
"releaseType" : "User"
},
"captureTime" : "2025-01-07 15:12:30.1013 +0000",
"codeSigningMonitor" : 1,
"incident" : "38B0B6A5-AC4B-438D-9ED5-DDD274E676C1",
"pid" : 51654,
"translated" : false,
"cpuType" : "ARM-64",
"roots_installed" : 0,
"bug_type" : "309",
"procLaunch" : "2025-01-07 15:12:30.0878 +0000",
"procStartAbsTime" : 12276109475883,
"procExitAbsTime" : 12276109794958,
"procName" : "python3.12",
"procPath" : "\/Users\/USER\/*\/python3.12",
"parentProc" : "python",
"parentPid" : 51646,
"coalitionName" : "com.github.wez.wezterm",
"crashReporterKey" : "5BB32C2E-55C1-DCA5-AE8B-3531EFE6FBEE",
"responsiblePid" : 804,
"responsibleProc" : "wezterm-gui",
"codeSigningID" : "-",
"codeSigningTeamID" : "",
"codeSigningFlags" : 570556961,
"codeSigningValidationCategory" : 10,
"codeSigningTrustLevel" : 4294967295,
"instructionByteStream" : {"beforePC":"fyMD1f17v6n9AwCRd+D\/l78DAJH9e8Go\/w9f1sADX9YQKYDSARAA1A==","atPC":"AwEAVH8jA9X9e7+p\/QMAkWzg\/5e\/AwCR\/XvBqP8PX9bAA1\/WcAqA0g=="},
"wakeTime" : 1751,
"sleepWakeUUID" : "92850A44-01E1-4036-91AC-9437DCBBCA46",
"sip" : "enabled",
"vmRegionInfo" : "0x104d60a8e is not in any region. Bytes after previous region: 2703 Bytes before following region: 128370\n REGION TYPE START - END [ VSIZE] PRT\/MAX SHRMOD REGION DETAIL\n MALLOC metadata 104d5c000-104d60000 [ 16K] rw-\/rwx SM=COW \n---> GAP OF 0x20000 BYTES\n __TEXT 104d80000-104d84000 [ 16K] r-x\/rwx SM=COW \/Users\/USER\/*\/_setproctitle.cpython-312-darwin.so",
"exception" : {"codes":"0x0000000000000001, 0x0000000104d60a8e","rawCodes":[1,4376103566],"type":"EXC_BAD_ACCESS","signal":"SIGSEGV","subtype":"KERN_INVALID_ADDRESS at 0x0000000104d60a8e"},
"termination" : {"flags":0,"code":11,"namespace":"SIGNAL","indicator":"Segmentation fault: 11","byProc":"python3.12","byPid":51654},
"vmregioninfo" : "0x104d60a8e is not in any region. Bytes after previous region: 2703 Bytes before following region: 128370\n REGION TYPE START - END [ VSIZE] PRT\/MAX SHRMOD REGION DETAIL\n MALLOC metadata 104d5c000-104d60000 [ 16K] rw-\/rwx SM=COW \n---> GAP OF 0x20000 BYTES\n __TEXT 104d80000-104d84000 [ 16K] r-x\/rwx SM=COW \/Users\/USER\/*\/_setproctitle.cpython-312-darwin.so",
"asi" : {"libsystem_c.dylib":["crashed on child side of fork pre-exec"]},
"extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
"faultingThread" : 0,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1895730265
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
```
task_sdk/tests/execution_time/test_supervisor.py::TestWatchedSubprocess::test_supervise_handles_deferred_task Fatal Python error: Segmentation fault
Current thread 0x00000001fb058f40 (most recent call first):
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/setproctitle/__init__.py", line 57 in <module>
File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 995 in exec_module
File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
File "/Users/ash/code/airflow/airflow/task_sdk/src/airflow/sdk/execution_time/task_runner.py", line 217 in startup
File "/Users/ash/code/airflow/airflow/task_sdk/src/airflow/sdk/execution_time/task_runner.py", line 339 in main
File "/Users/ash/code/airflow/airflow/task_sdk/src/airflow/sdk/execution_time/supervisor.py", line 135 in _subprocess_main
File "/Users/ash/code/airflow/airflow/task_sdk/src/airflow/sdk/execution_time/supervisor.py", line 256 in _fork_main
File "/Users/ash/code/airflow/airflow/task_sdk/src/airflow/sdk/execution_time/supervisor.py", line 349 in start
File "/Users/ash/code/airflow/airflow/task_sdk/src/airflow/sdk/execution_time/supervisor.py", line 899 in supervise
File "/Users/ash/code/airflow/airflow/task_sdk/tests/execution_time/test_supervisor.py", line 295 in test_supervise_handles_deferred_task
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/python.py", line 1627 in runtest
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 242 in <lambda>
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 241 in call_and_report
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 132 in runtestprotocol
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/main.py", line 337 in _main
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/main.py", line 283 in wrap_session
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 175 in main
File "/Users/ash/code/airflow/airflow/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 201 in console_main
File "/Users/ash/code/airflow/airflow/.venv/bin/pytest", line 8 in <module>
Extension modules: _time_machine, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, sqlalchemy.cimmutabledict, sqlalchemy.cprocessors, sqlalchemy.cresultproxy, yaml._yaml, _cffi_backend, lazy_object_proxy.cext, markupsafe._speedups, psutil._psutil_osx, psutil._psutil_posix, msgspec._core, psycopg2._psycopg, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, setproctitle._setproctitle (total: 24)
FAILED
```
One such error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "gershnik (via GitHub)" <gi...@apache.org>.
gershnik commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1898576576
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@ashb thats where the setproctitle implementation was taken from 🙂
I am not going to be near a computer until next week so cannot comment meaningfully but one thing that would be helpful is to get a _native_ stack trace for the crash. It can usually be found in Console app once Python crashes.
My fist though would be that this is yet another interaction with fork() but strange that it is not deterministic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "gershnik (via GitHub)" <gi...@apache.org>.
gershnik commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1899941148
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@ashb I've run `pytest task_sdk` (clean clone from master branch) hundreds of times on both Intel and Apple Silicone Macs (macOS 15.1.1 and 15.0.1 respectively) but it didn't crash for me. Obviously something is different with my setup but what? Puzzled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1906949527
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
So this one is slightly odd. It's def triggerable 100% of the time for me.
Acording to python `threading` module there is only a single thread at the point of calling `os.fork`.
The odd thing here is that if I import `setproctitle` eagerly before fork, then the SIGSEGV goes away, but Py 3.12 now starts complaining about "This process (pid=96305) is multi-threaded, use of fork() may lead to deadlocks in the child."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1895731135
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
Trying with `uv pip install git+https://github.com/dvarrazzo/py-setproctitle` and I get the exact same behaviour fwiw.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1898548678
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@gershnik My plan if I get some spare cycles was to compare the current impl with the Node/libuv implementation: https://github.com/nodejs/node/blob/9cef7b84b1f6b1e116afaad945f31b368e8917e2/deps/uv/src/unix/darwin-proctitle.c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "gershnik (via GitHub)" <gi...@apache.org>.
gershnik commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1905810569
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@ashb Thank you, this is super helpful! (I am still unable to reproduce this even once - just tried it again after updating macOS)
With regards to threads, setproctitle doesn't by itself create any threads. Apple frameworks do so internally for their XPC with Launch Services but those sit dormant unless functionality that is using them is invoked. Also see below.
The crash happens in a child process post-fork on a main thread, early during setproctitle initialization in [CFBundleGetBundleWithIdentifier](https://developer.apple.com/documentation/corefoundation/1537139-cfbundlegetbundlewithidentifier?language=objc) call. It is called by setproctitle with a static argument[^1]:
```cpp
CFBundleGetBundleWithIdentifier(CFSTR("com.apple.LaunchServices"))
```
so it references no caller-supplied memory that can become invalid somehow. Thus, it is the _internal_ memory of CoreFoundation that is somehow corrupted at the time of this call. In other words the crash is "impossible" unless CoreFoundation itself is in a broken state.
Also note that any threads Apple might create hasn't been started yet - this call happens long before such functionality is invoked.
All of this, combined with the fact that the crash is very non-deterministic suggests that setproctitle is a victim here of something (potentially itself) using Apple APIs on another thread in parallel with fork.
So the question is whether this is what is going on. Are there any calls to to setproctitle (including importing it) or any other Apple-using library in the parent process that can happen in parallel with fork?
[Update]
@potiuk - just realized that your comment indicates that this is actually a known issue that has other manifestations, correct?
[^1]: `CFSTR` is an Apple macro to produce a statically allocated CFString constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1905602466
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
@potiuk Setproctitle creates a thread internally at import time on macos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1894996288
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
I am in contact with setproctitle maintainer during the "Airflow Beach Cleaning" project. I can ask him to comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1895728978
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
`pytest task_sdk` (locally, not breeze) would trigger it about 10-25% of the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1895019882
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
After short discussion with @dvarrazzo - it's likely this https://github.com/dvarrazzo/py-setproctitle/pull/144 is going to fix it (unreleased yet).
It would be great though to get some more details about those segfaults @jaketf @ashb when you see them happening again ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1905604226
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
> @ashb I've run `pytest task_sdk` (clean clone from master branch) hundreds of times on both Intel and Apple Silicone Macs (macOS 15.1.1 and 15.0.1 respectively) but it didn't crash for me. Obviously something is different with my setup but what? Puzzled.
That's odd.
I can now almost 100% reproduce this error. M2 Macbook pro here on Sonoma 14.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] skip setproctitle in `task_runner` on Mac OS [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #45124:
URL: https://github.com/apache/airflow/pull/45124#discussion_r1905604226
##########
airflow/dag_processing/manager.py:
##########
@@ -181,7 +180,15 @@ def _run_processor_manager(
# to iterate the child processes
set_new_process_group()
- setproctitle("airflow scheduler -- DagFileProcessorManager")
+
+ # setproctitle causes issue on Mac OS: https://github.com/benoitc/gunicorn/issues/3021
+ os_type = sys.platform
+ if os_type == "darwin":
+ log.info("Mac OS detected, skipping setproctitle")
+ else:
+ from setproctitle import setproctitle
+ setproctitle("airflow scheduler -- DagFileProcessorManager")
Review Comment:
> @ashb I've run `pytest task_sdk` (clean clone from master branch) hundreds of times on both Intel and Apple Silicone Macs (macOS 15.1.1 and 15.0.1 respectively) but it didn't crash for me. Obviously something is different with my setup but what? Puzzled.
@gershnik That's odd.
I can now almost 100% reproduce this error. M2 Macbook pro here on Sonoma 14.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org