You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/03 11:23:13 UTC

[GitHub] [airflow] uranusjr opened a new pull request #20011: Type-annotate SkipMixin and BaseXCom

uranusjr opened a new pull request #20011:
URL: https://github.com/apache/airflow/pull/20011


   Extracted from #19825, these are required to make Mypy pass checking the two files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r761936426



##########
File path: airflow/models/xcom.py
##########
@@ -376,4 +505,7 @@ def resolve_xcom_backend():
     return BaseXCom
 
 
-XCom = resolve_xcom_backend()
+if TYPE_CHECKING:
+    XCom = BaseXCom  # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+else:
+    XCom = resolve_xcom_backend()

Review comment:
       Every so often I've tried upgrading sphinx-autoapi, but last _n_ times I tried there was a bug that meant it didn't build docs on a more recent version :(




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr merged pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr merged pull request #20011:
URL: https://github.com/apache/airflow/pull/20011


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r763000669



##########
File path: airflow/models/xcom.py
##########
@@ -79,144 +80,247 @@ def init_on_load(self):
     def __repr__(self):
         return f'<XCom "{self.key}" ({self.task_id} @ {self.execution_date})>'
 
+    @overload
     @classmethod
-    @provide_session
-    def set(cls, key, value, task_id, dag_id, execution_date=None, run_id=None, session=None):
+    def set(
+        cls,
+        key: str,
+        value: Any,
+        *,
+        dag_id: str,
+        task_id: str,
+        run_id: str,
+        session: Optional[Session] = None,
+    ) -> None:
+        """Store an XCom value.
+
+        A deprecated form of this function accepts ``execution_date`` instead of
+        ``run_id``. The two arguments are mutually exclusive.
+
+        :param key: Key to store the XCom.
+        :param value: XCom value to store.
+        :param dag_id: DAG ID.
+        :param task_id: Task ID.
+        :param run_id: DAG run ID for the task.
+        :param session: Database session. If not given, a new session will be
+            created for this function.
+        :type session: sqlalchemy.orm.session.Session
         """
-        Store an XCom value.
 
-        :return: None
-        """
+    @overload
+    @classmethod
+    def set(
+        cls,
+        key: str,
+        value: Any,
+        task_id: str,
+        dag_id: str,
+        execution_date: datetime.datetime,
+        session: Optional[Session] = None,
+    ) -> None:
+        """:sphinx-autoapi-skip:"""
+
+    @classmethod
+    @provide_session
+    def set(
+        cls,
+        key: str,
+        value: Any,
+        task_id: str,
+        dag_id: str,
+        execution_date: Optional[datetime.datetime] = None,
+        *,
+        run_id: Optional[str] = None,
+        session: Session,

Review comment:
       Ditto




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r761916792



##########
File path: airflow/models/xcom.py
##########
@@ -376,4 +505,7 @@ def resolve_xcom_backend():
     return BaseXCom
 
 
-XCom = resolve_xcom_backend()
+if TYPE_CHECKING:
+    XCom = BaseXCom  # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+else:
+    XCom = resolve_xcom_backend()

Review comment:
       Well actually it does support it since last year, but we’re pinning an extremely old version.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r761909724



##########
File path: airflow/models/xcom.py
##########
@@ -376,4 +505,7 @@ def resolve_xcom_backend():
     return BaseXCom
 
 
-XCom = resolve_xcom_backend()
+if TYPE_CHECKING:
+    XCom = BaseXCom  # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+else:
+    XCom = resolve_xcom_backend()

Review comment:
       I think this might be causing docs to fail too, as it's somehow finding two places for all the methods on XCom.
   
   (I'm guessing as to cause, doc build is failing though)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#issuecomment-987636734


   Rebasing (again) to hopefully fix the last of test failures.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#issuecomment-985494002


   Tests are failing with "Object of type bytes is not JSON serializable" on trying to send XCom over the API.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r761976058



##########
File path: airflow/models/xcom.py
##########
@@ -376,4 +505,7 @@ def resolve_xcom_backend():
     return BaseXCom
 
 
-XCom = resolve_xcom_backend()
+if TYPE_CHECKING:
+    XCom = BaseXCom  # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+else:
+    XCom = resolve_xcom_backend()

Review comment:
       Why are we using autoapi in the first place? Do we have any non-Python APIs that can’t be documented with sphinx.ext.autodoc?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r761872738



##########
File path: airflow/models/skipmixin.py
##########
@@ -17,7 +17,9 @@
 # under the License.
 
 import warnings
-from typing import TYPE_CHECKING, Iterable, Union
+from typing import TYPE_CHECKING, Iterable, Optional, Sequence, Union
+
+import pendulum

Review comment:
       Since we already have `if TYPE_CHECKING` on L30 move this to that block?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#issuecomment-985446465


   The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r762764645



##########
File path: airflow/models/xcom.py
##########
@@ -99,29 +137,63 @@ def set(cls, key, value, task_id, dag_id, execution_date=None, run_id=None, sess
 
         value = XCom.serialize_value(value)
 
-        # remove any duplicate XComs
+        # Remove duplicate XComs and insert a new one.
         session.query(cls).filter(
-            cls.key == key, cls.execution_date == execution_date, cls.task_id == task_id, cls.dag_id == dag_id
+            cls.key == key,
+            cls.execution_date == execution_date,
+            cls.task_id == task_id,
+            cls.dag_id == dag_id,
         ).delete()
-
+        new = cast(Any, cls)(  # Work around Mypy complaining model not defining '__init__'.
+            key=key,
+            value=cls.serialize_value(value),

Review comment:
       Ah I see, wrong copy-paste. I merged line 100/138 above to this in the other PR, but forgot to delete that line when copying the new implementation.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r763000313



##########
File path: airflow/models/skipmixin.py
##########
@@ -66,9 +67,10 @@ def _set_state_to_skipped(self, dag_run: "DagRun", tasks: "Iterable[BaseOperator
     def skip(
         self,
         dag_run: "DagRun",
-        execution_date: "timezone.DateTime",
-        tasks: "Iterable[BaseOperator]",
-        session: "Session" = None,
+        execution_date: "DateTime",
+        tasks: Sequence["BaseOperator"],
+        *,
+        session: "Session",

Review comment:
       We probably shouldn't make session a kw-only arg, as that _might_ break someone's custom operator :( 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r761881509



##########
File path: airflow/models/skipmixin.py
##########
@@ -17,7 +17,9 @@
 # under the License.
 
 import warnings
-from typing import TYPE_CHECKING, Iterable, Union
+from typing import TYPE_CHECKING, Iterable, Optional, Sequence, Union
+
+import pendulum

Review comment:
       Moved




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r761914076



##########
File path: airflow/models/xcom.py
##########
@@ -99,29 +137,63 @@ def set(cls, key, value, task_id, dag_id, execution_date=None, run_id=None, sess
 
         value = XCom.serialize_value(value)
 
-        # remove any duplicate XComs
+        # Remove duplicate XComs and insert a new one.
         session.query(cls).filter(
-            cls.key == key, cls.execution_date == execution_date, cls.task_id == task_id, cls.dag_id == dag_id
+            cls.key == key,
+            cls.execution_date == execution_date,
+            cls.task_id == task_id,
+            cls.dag_id == dag_id,
         ).delete()
-
+        new = cast(Any, cls)(  # Work around Mypy complaining model not defining '__init__'.
+            key=key,
+            value=cls.serialize_value(value),

Review comment:
       Hmm not sure how this got here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#issuecomment-987745926


   ✅  🎉 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r762628786



##########
File path: airflow/models/xcom.py
##########
@@ -376,4 +505,7 @@ def resolve_xcom_backend():
     return BaseXCom
 
 
-XCom = resolve_xcom_backend()
+if TYPE_CHECKING:
+    XCom = BaseXCom  # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+else:
+    XCom = resolve_xcom_backend()

Review comment:
       I think this is question mostly to @mik-laj who knows more of the context. But I share the same sentiment and experience as @ashb - any time I tried I backtracked so far. And yea - we are using extremely old version.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#issuecomment-986552930


   Wow autoapi 1.0 is so fucking ancient even my workaround does not work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#issuecomment-986653528


   I added a hack to allow manually telling AutoAPI to ignore any code construct if the docstring contains `:sphinx-autoapi-skip:`. This is probably the best thing we can do, short of fixing overloads entirely.
   
   Note that I chose to document the _desired_ interface that uses `run_id`, instead of the _actual_ interface that accepts both `run_id` and `execution_date`, because, well, we don’t want people to use `execution_date` if possible 🙂 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#issuecomment-987276180


   I have just rebased this PR after the fix on main.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r761915807



##########
File path: airflow/models/xcom.py
##########
@@ -376,4 +505,7 @@ def resolve_xcom_backend():
     return BaseXCom
 
 
-XCom = resolve_xcom_backend()
+if TYPE_CHECKING:
+    XCom = BaseXCom  # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+else:
+    XCom = resolve_xcom_backend()

Review comment:
       I think this is fine, the cause is actually the overloads:
   
   ```
   WARNING: duplicate object description of airflow.models.xcom.BaseXCom.clear, other instance in _api/airflow/models/xcom/index, use :noindex: for one of them
   
   File path: apache-airflow/_api/airflow/models/xcom/index.rst (215)
   
    210 |    .. classmethod:: clear(cls, execution_date: pendulum.DateTime, dag_id: str, task_id: str, session: Optional[Session] = None)
    211 | 
    212 | 
    213 | 
    214 |    
   >215 |    .. classmethod:: clear(cls, execution_date: Optional = None, dag_id: Optional = None, task_id: Optional = None, *, run_id: Optional = None, session: Session)
    216 | 
    217 |       Clears all XCom data from the database for the task instance
    218 | 
    219 |       ``run_id`` and ``execution_date`` are mutually exclusive.
   ```
   
   I think generated by sphinx-autoapi. I’d expect it to support this _years_ old feature; it’s very disappoiting.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20011: Type-annotate SkipMixin and BaseXCom

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20011:
URL: https://github.com/apache/airflow/pull/20011#discussion_r761908694



##########
File path: airflow/models/xcom.py
##########
@@ -99,29 +137,63 @@ def set(cls, key, value, task_id, dag_id, execution_date=None, run_id=None, sess
 
         value = XCom.serialize_value(value)
 
-        # remove any duplicate XComs
+        # Remove duplicate XComs and insert a new one.
         session.query(cls).filter(
-            cls.key == key, cls.execution_date == execution_date, cls.task_id == task_id, cls.dag_id == dag_id
+            cls.key == key,
+            cls.execution_date == execution_date,
+            cls.task_id == task_id,
+            cls.dag_id == dag_id,
         ).delete()
-
+        new = cast(Any, cls)(  # Work around Mypy complaining model not defining '__init__'.
+            key=key,
+            value=cls.serialize_value(value),

Review comment:
       This is different -- before it was just doing `value=value`.
   
   ```suggestion
               value=value,
   ```
   
   I think?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org