You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jcobb-healx (via GitHub)" <gi...@apache.org> on 2023/02/03 14:40:06 UTC

[GitHub] [airflow] jcobb-healx opened a new issue, #29358: Cannot use TypedDict object when defining params

jcobb-healx opened a new issue, #29358:
URL: https://github.com/apache/airflow/issues/29358

   ### Apache Airflow version
   
   2.5.1
   
   ### What happened
   
   Context: I am attempting to use [TypedDict](https://docs.python.org/3/library/typing.html#typing.TypedDict) objects to maintain the keys used in DAG params in a single place, and check for key names across multiple DAGs that use the params.
   This raises an error with `mypy` as `params` expects an `Optional[Dict]`. Due to the invariance of `Dict`, this does not accept `TypedDict` objects.
   
   What happened: I passed a `TypedDict` to the `params` arg of `DAG` and got a TypeError.
   
   ### What you think should happen instead
   
   `TypedDict` objects should be accepted by `DAG`, which should accept `Optional[Mapping[str, Any]]`. 
   
   Unless I'm mistaken, `params` are converted to a `ParamsDict` class and therefore the appropriate type hint is a generic `Mapping` type.
   
   ### How to reproduce
   
   Steps to reproduce
   
   ```Python
   from typing import TypedDict
   
   from airflow import DAG
   from airflow.models import Param
   
   class ParamsTypedDict(TypedDict):
       str_param: Param
   
   params: ParamsTypedDict = {
       "str_param": Param("", type="str")
   }
   
   with DAG(
       dag_id="mypy-error-dag",
       # The line below raises a mypy error
       # Argument "params" to "DAG" has incompatible type "ParamsTypedDict"; expected "Optional[Dict[Any, Any]]"  [arg-type]
       params=params,  
   ) as dag:
       pass
   ```
   
   ### Operating System
   
   Amazon Linux
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pierrejeambrun commented on issue #29358: Cannot use TypedDict object when defining params

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on issue #29358:
URL: https://github.com/apache/airflow/issues/29358#issuecomment-1445166734

   @jcobb-healx Are you still working on this ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jcobb-healx commented on issue #29358: Cannot use TypedDict object when defining params

Posted by "jcobb-healx (via GitHub)" <gi...@apache.org>.
jcobb-healx commented on issue #29358:
URL: https://github.com/apache/airflow/issues/29358#issuecomment-1446280306

   @pierrejeambrun I've made the changes to use `MutableMapping` so that passing in a `ParamsDict` object works as expected. I've not changed anything that would allow my original case to work, so I've made no mention of `TypedDict` in the PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29358: Cannot use TypedDict object when defining params

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29358:
URL: https://github.com/apache/airflow/issues/29358#issuecomment-1415962009

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #29358: Cannot use TypedDict object when defining params

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29358:
URL: https://github.com/apache/airflow/issues/29358#issuecomment-1416062014

   Feel free to work on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jcobb-healx commented on issue #29358: Cannot use TypedDict object when defining params

Posted by "jcobb-healx (via GitHub)" <gi...@apache.org>.
jcobb-healx commented on issue #29358:
URL: https://github.com/apache/airflow/issues/29358#issuecomment-1446254058

   @pierrejeambrun Sorry, I hadn't gotten around to looking at this. Spent a bit of time on this today, and it looks like converting the type hint to `Mapping` isn't right, as the `params` are merged with the default params in an [`update` call](https://github.com/apache/airflow/blob/a5adb87ab4ee537eb37ef31aba755b40f6f29a1e/airflow/models/dag.py#L430), which isn't consistent with `TypedDict` behaviour.
   Given the earlier `mypy` discussion, it probably makes sense to use `MutableMapping` to allow generic inputs, and let users perform casts, i.e. _not_ accept `TypedDict` objects.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jcobb-healx commented on issue #29358: Cannot use TypedDict object when defining params

Posted by "jcobb-healx (via GitHub)" <gi...@apache.org>.
jcobb-healx commented on issue #29358:
URL: https://github.com/apache/airflow/issues/29358#issuecomment-1416034175

   Relevant mypy discussion on passing `TypedDict` as args annotated with `Dict`: https://github.com/python/mypy/issues/4976


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #29358: Cannot use TypedDict object when defining params

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29358: Cannot use TypedDict object when defining params
URL: https://github.com/apache/airflow/issues/29358


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org