[GitHub] [airflow] Calder-Ty commented on issue #17045: "Failed to load task run error" occur in "on_failure_callback" after upgrading from 2.0.0 to 2.1.2

[GitBox <gi...@apache.org>]

Calder-Ty commented on issue #17045:
URL: https://github.com/apache/airflow/issues/17045#issuecomment-908607130


   **Bottom Line Up Front**: I had a similar issue, and found that the cause was a third party packages code that did not implement exceptions that could be loaded from a picklefile. This is the most likely cause of the problem.
   
   For me, switching to Airflows official helm wasn't an option (not running Kubernetes). I did some digging around and found that this message is spawned when trying to load an original exception from a pickle file. 
   https://github.com/apache/airflow/blob/9922287a4f9f70b57635b04436ddc4cfca0e84d2/airflow/models/taskinstance.py#L136-L147 
   
   Unfortunately the causing error message is not logged. I applied this patch to my instance of airflow to see what error was causing the pickle.loads to fail.
   ```
   @@ -115,7 +115,10 @@
            return None
        try:
            return pickle.loads(data)
   -    except Exception:  # pylint: disable=broad-except
   +    except Exception as e:  # pylint: disable=broad-except
   +        log.exception("Failed to load the exception! Oh NO!!! Here is why")
   +        log.exception(e.msg)
   +        log.exception(e)
            return "Failed to load task run error"
   ```
   
   I found that the error was being caused  because the third party tooling I was using had an error that could be pickled, but then not loaded back. [This is a common issue with custom errors](https://stackoverflow.com/questions/41808912/cannot-unpickle-exception-subclass) in python. I'm guessing that something similar is happening in your case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org