Re: [I] Airflow 3: cannot run a `PythonVirtualenvOperator` with `pendulum<3` [airflow]

["sjyangkevin (via GitHub)" <gi...@apache.org>]

sjyangkevin commented on issue #50752:
URL: https://github.com/apache/airflow/issues/50752#issuecomment-2994854773

   Hi,
   
   I have a deeper look into the issue recently and found some challenges about how to resolve the issue. I would appreciate if I could get some help, or if someone had better knowledge could take it. Thanks.
   
   In the operator, we are using three serialization library including "pickle", "cloudpickle", and "dill", and "pickle" is the default one. I think we don't have control over the internal mechanism on how these libraries serialize and deserialize those objects.
   
   I tried to convert the `pendulum.DateTime` objects, as shown below, into string type before these being serialized by any of the above library in `_write_args`. However, I got another error `AttributeError: module 'builtins' has no attribute '_unpickle'`.  
   
   * "data_interval_end"
   * "data_interval_start"
   * "logical_date"
   * "prev_data_interval_end_success"
   * "prev_data_interval_start_success"
   * "prev_start_date_success"
   * "prev_end_date_success"
   
   Below is my thoughts on how this can be resolved, but it didn't work out.
   * Identify the version of pendulum in the "main" (Airflow) environment, and the version of pendulum in the virtual environment (through requirement list)
   * If the `found_pendulum` and the `pendulum<3` in the virtual environment, and `pendulum>=3` in the Airflow environment, we need to patch it with some preprocessing to the objects above before serialization (e.g., convert to string)
   * In the Python template, implement a process to recover those objects back to `pendulum.DateTime`, and when send it back, patch again to make it serializable and can be recovered to pendulum 3 compatible objects.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org