[GitHub] [airflow] uranusjr opened a new pull request #19830: Avoid using Proxy in subscript type alias

[GitBox <gi...@apache.org>]

uranusjr opened a new pull request #19830:
URL: https://github.com/apache/airflow/pull/19830


   This should fix main.
   
   Python's pickle module (or typing?) seems to have a bug when `lazy_object_proxy.Proxy` is used in a subscript type (such as Union). `pickle` would somehow incorrectly try to look up the Proxy type in
   `builtin` during pickling, causing an exception.
   
   Since this global variable being pickled is only a type alias and not really functionally significant, we can work around the bug by simply not introducing that alias in the first place.
   
   I’ll try to come up with a more minimal example to reproduce the issue and report this to CPython. :)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #19830: Avoid using Proxy in subscript type alias

[GitBox <gi...@apache.org>]

potiuk merged pull request #19830:
URL: https://github.com/apache/airflow/pull/19830


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #19830: Avoid using Proxy in subscript type alias

[GitBox <gi...@apache.org>]

uranusjr commented on pull request #19830:
URL: https://github.com/apache/airflow/pull/19830#issuecomment-979439563


   Managed to reduce the problem and it became obvious. To properly “fake” an object, `lazy-object-proxy` uses a metaclass to “fake” things so `type(proxy).__module__` returns a “correct” value. But when the Proxy type is used directly, `Proxy.__module__` ends up not referencing the correct module bcause there is not a value to wrap. The behaviour actually depends on how `lazy-object-proxy` is implemented; the C version would just return NULL, which represents the builtin namespace, resulting in the cryptic error message.
   
   The Python version is even worse because it implements `Proxy.__module__` as a property, which cannot even be handled and would fail with something like
   
   ```
   _pickle.PicklingError: Can't pickle <class 'Proxy'>: import of module <property object at 0x7f7b1d3d4770> failed
   ```
   
   Either way, the issue is in `lazy-object-proxy`, and is a very nasty abstraction leakage. The C version is not actually too difficult to fix, but the Python version is not, and it does not make sense to only fix the issue in one implementation. So I guess the best we can do is just… avoid it, like how this PR does.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on pull request #19830: Avoid using Proxy in subscript type alias

[GitBox <gi...@apache.org>]

uranusjr commented on pull request #19830:
URL: https://github.com/apache/airflow/pull/19830#issuecomment-979421704


   On further investigation, the `Union` part is irrelevant[^1]; I can trivially trigger a crash with
   
   ```python
   import io
   import pickle
   from lazy_object_proxy import Proxy
   pickle.dump(Proxy, io.BytesIO())
   # Can't pickle <class 'Proxy'>: attribute lookup Proxy on builtins failed
   ```
   
   Next is to decide whether `lazy_object_proxy` or CPython is responsible for this.
   
   [^1]: Although Union did _trigger_ the problem for Airflow; `pickle` has no problem pickling the `lazy_object_proxy` module itself; it only fails when `Proxy` is referenced _as a type_ outside of the `lazy_object_proxy` module.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org