You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2024/08/05 08:31:04 UTC

[PR] Swallow exception when mini-scheduling raises an exception [airflow]

potiuk opened a new pull request, #41260:
URL: https://github.com/apache/airflow/pull/41260

   When mini-scheduler raises an exception, it has a bit weird side effect - the task succeeds but it is seen as failed and scheduler gets confused. Also flower celery worker in this case shows an error.
   
   This happens for example when DAG contains non-serializable tasks.
   
   This PR swallows any exceptions raised in mini-scheduler and simply logs them as error rather than fail the process. Mini-scheduler is generally optional and we are also already sometimes skipping it already so occasional skipping is not a big problem.
   
   Fixes: #39717
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41260:
URL: https://github.com/apache/airflow/pull/41260#discussion_r1704003495


##########
airflow/models/taskinstance.py:
##########
@@ -3892,9 +3892,12 @@ def schedule_downstream_tasks(self, session: Session = NEW_SESSION, max_tis_per_
 
         :meta: private
         """
-        return TaskInstance._schedule_downstream_tasks(
-            ti=self, session=session, max_tis_per_query=max_tis_per_query
-        )
+        try:
+            return TaskInstance._schedule_downstream_tasks(
+                ti=self, session=session, max_tis_per_query=max_tis_per_query
+            )
+        except Exception:
+            self.log.exception("Error scheduling downstream tasks. Skipping it")

Review Comment:
   I **think** it's still useful to know that it's happening - the main thing it currently struggles with is the exit code that is now 0. And it was quite confusing because task succeeded, but the process failed.
   
   I imagine this log will provide some useful information - for example if we just silently discard it in https://github.com/apache/airflow/issues/39717  - we would never have known that the problem is with serializing. Which **might** be something theat the user can do something about - we have no idea, because there might be varioue reasons for the exception.
   
   I believe it is quite good to explicitly tell "something happen" - and while yes - we do not know exactly what the user can do to avoid it, it will at least give the user a chance to understand and fix it (or maybe submit an issue / discussion here and get help.
   
   I am generally very, very cautious about totally ignoring and silencing such exceptions - because we have no idea what's going on. In this case we decided that we won't do a confusing thing - because mini-scheduler is purely optional we can just not fail the process, but having a log is still useful.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #41260:
URL: https://github.com/apache/airflow/pull/41260


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41260:
URL: https://github.com/apache/airflow/pull/41260#discussion_r1705044857


##########
airflow/models/taskinstance.py:
##########
@@ -3892,9 +3892,17 @@ def schedule_downstream_tasks(self, session: Session = NEW_SESSION, max_tis_per_
 
         :meta: private
         """
-        return TaskInstance._schedule_downstream_tasks(
-            ti=self, session=session, max_tis_per_query=max_tis_per_query
-        )
+        try:
+            return TaskInstance._schedule_downstream_tasks(
+                ti=self, session=session, max_tis_per_query=max_tis_per_query
+            )
+        except Exception:
+            self.log.exception(
+                "Error scheduling downstream tasks. Skipping it as this is entirely optional optimisation. "
+                "There might be various reasons for it, please take a look at the stack trace to figure "
+                "out if the root cause can be diagnosed and fixed. See the issue "
+                "https://github.com/apache/airflow/issues/39717 for details and example problem."

Review Comment:
   Sure. Good idea as well. Added some more things.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41260:
URL: https://github.com/apache/airflow/pull/41260#issuecomment-2497412418

   Likely unrelated. I suggest to start separate Github discussion an describe it there in details, explaining your deployment and circumstances what exactly happened. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #41260:
URL: https://github.com/apache/airflow/pull/41260#discussion_r1704559495


##########
airflow/models/taskinstance.py:
##########
@@ -3892,9 +3892,12 @@ def schedule_downstream_tasks(self, session: Session = NEW_SESSION, max_tis_per_
 
         :meta: private
         """
-        return TaskInstance._schedule_downstream_tasks(
-            ti=self, session=session, max_tis_per_query=max_tis_per_query
-        )
+        try:
+            return TaskInstance._schedule_downstream_tasks(
+                ti=self, session=session, max_tis_per_query=max_tis_per_query
+            )
+        except Exception:
+            self.log.exception("Error scheduling downstream tasks. Skipping it")

Review Comment:
   > I believe it is quite good to explicitly tell "something happen" - and while yes - we do not know exactly what the user can do to avoid it, it will at least give the user a chance to understand and fix it (or maybe submit an issue / discussion here and get help.
   
   Then lets say this explicitly in the message. I think we can just say that it can be caused by several reasons and link to the relevant discussion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41260:
URL: https://github.com/apache/airflow/pull/41260#discussion_r1704650941


##########
airflow/models/taskinstance.py:
##########
@@ -3892,9 +3892,12 @@ def schedule_downstream_tasks(self, session: Session = NEW_SESSION, max_tis_per_
 
         :meta: private
         """
-        return TaskInstance._schedule_downstream_tasks(
-            ti=self, session=session, max_tis_per_query=max_tis_per_query
-        )
+        try:
+            return TaskInstance._schedule_downstream_tasks(
+                ti=self, session=session, max_tis_per_query=max_tis_per_query
+            )
+        except Exception:
+            self.log.exception("Error scheduling downstream tasks. Skipping it")

Review Comment:
   @ephraimbuddy @eladkal -> see if you like it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #41260:
URL: https://github.com/apache/airflow/pull/41260#discussion_r1703934310


##########
airflow/models/taskinstance.py:
##########
@@ -3892,9 +3892,12 @@ def schedule_downstream_tasks(self, session: Session = NEW_SESSION, max_tis_per_
 
         :meta: private
         """
-        return TaskInstance._schedule_downstream_tasks(
-            ti=self, session=session, max_tis_per_query=max_tis_per_query
-        )
+        try:
+            return TaskInstance._schedule_downstream_tasks(
+                ti=self, session=session, max_tis_per_query=max_tis_per_query
+            )
+        except Exception:
+            self.log.exception("Error scheduling downstream tasks. Skipping it")

Review Comment:
   Does this call for action from user side?
   The thing about logging such exceptions is that users want to solve the underlying issue but from the description it sounds like there is no real action item to avoid it? Should we leave hints in the message (for example consider disable mini scheduler)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41260:
URL: https://github.com/apache/airflow/pull/41260#discussion_r1704599731


##########
airflow/models/taskinstance.py:
##########
@@ -3892,9 +3892,12 @@ def schedule_downstream_tasks(self, session: Session = NEW_SESSION, max_tis_per_
 
         :meta: private
         """
-        return TaskInstance._schedule_downstream_tasks(
-            ti=self, session=session, max_tis_per_query=max_tis_per_query
-        )
+        try:
+            return TaskInstance._schedule_downstream_tasks(
+                ti=self, session=session, max_tis_per_query=max_tis_per_query
+            )
+        except Exception:
+            self.log.exception("Error scheduling downstream tasks. Skipping it")

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41260:
URL: https://github.com/apache/airflow/pull/41260#discussion_r1704594490


##########
airflow/models/taskinstance.py:
##########
@@ -3892,9 +3892,12 @@ def schedule_downstream_tasks(self, session: Session = NEW_SESSION, max_tis_per_
 
         :meta: private
         """
-        return TaskInstance._schedule_downstream_tasks(
-            ti=self, session=session, max_tis_per_query=max_tis_per_query
-        )
+        try:
+            return TaskInstance._schedule_downstream_tasks(
+                ti=self, session=session, max_tis_per_query=max_tis_per_query
+            )
+        except Exception:
+            self.log.exception("Error scheduling downstream tasks. Skipping it")

Review Comment:
   Good idea.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "boris-tumbev (via GitHub)" <gi...@apache.org>.
boris-tumbev commented on PR #41260:
URL: https://github.com/apache/airflow/pull/41260#issuecomment-2497138881

   Hey Team, we had the problem that is suppose to be fixed in this PR, however recently on airflow 2.10.2 we had this:
   ![image](https://github.com/user-attachments/assets/56d921f6-d178-45d3-be65-6e5451e44da5)
   Task was success then marked as failed twice, can it be connected to the same problem ? From the UI i can see the log only for the second try :thinking: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #41260:
URL: https://github.com/apache/airflow/pull/41260#discussion_r1703995866


##########
airflow/models/taskinstance.py:
##########
@@ -3892,9 +3892,12 @@ def schedule_downstream_tasks(self, session: Session = NEW_SESSION, max_tis_per_
 
         :meta: private
         """
-        return TaskInstance._schedule_downstream_tasks(
-            ti=self, session=session, max_tis_per_query=max_tis_per_query
-        )
+        try:
+            return TaskInstance._schedule_downstream_tasks(
+                ti=self, session=session, max_tis_per_query=max_tis_per_query
+            )
+        except Exception:
+            self.log.exception("Error scheduling downstream tasks. Skipping it")

Review Comment:
   Yeah, this could confuse the users. We could pass and not log anything since the users can't do anything about it, and the scheduler will still schedule the given task. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Swallow exception when mini-scheduling raises an exception [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #41260:
URL: https://github.com/apache/airflow/pull/41260#discussion_r1704937003


##########
airflow/models/taskinstance.py:
##########
@@ -3892,9 +3892,17 @@ def schedule_downstream_tasks(self, session: Session = NEW_SESSION, max_tis_per_
 
         :meta: private
         """
-        return TaskInstance._schedule_downstream_tasks(
-            ti=self, session=session, max_tis_per_query=max_tis_per_query
-        )
+        try:
+            return TaskInstance._schedule_downstream_tasks(
+                ti=self, session=session, max_tis_per_query=max_tis_per_query
+            )
+        except Exception:
+            self.log.exception(
+                "Error scheduling downstream tasks. Skipping it as this is entirely optional optimisation. "
+                "There might be various reasons for it, please take a look at the stack trace to figure "
+                "out if the root cause can be diagnosed and fixed. See the issue "
+                "https://github.com/apache/airflow/issues/39717 for details and example problem."

Review Comment:
   ```suggestion
                   "https://github.com/apache/airflow/issues/39717 for details and an example problem."
   ```
   
   It may also be a good idea to tell users what to do after gettting this. Just ignore it? (What can I do if it’s noisy?) Post a comment to the issue or not?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org