You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/08 16:57:20 UTC
[GitHub] [airflow] andrewgodwin opened a new pull request #15285: Fixed #9387: Add State types for tasks and DAGs
andrewgodwin opened a new pull request #15285:
URL: https://github.com/apache/airflow/pull/15285
For unfortunate reasons, we can't use Enums (changes the ultimate type of the values from str to EnumValue, so a lot of stuff breaks) and we can't re-use a tuple/iterable of values as a type declaration, so this ends up being the way it has to be done.
I'm not super keen on this, but wanted to present it as one of three possible solutions, the others being:
* Switching TaskState and DagState to _actually_ be an enum, and have to go rewrite a large chunk of core code and potentially break compatability with many providers since it'll change the values of all the states to be of type `EnumValue`.
* Closing issue #9387 as "we're not going to do this now"
This PR does the main type addition and a few other type annotations around the codebase; it's not exhaustive, since I'm not yet sure we'll go this route, but if we do like it I'll probably hunt down some more places to annotate before finally merging.
closes: #9387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-815994439
> Switching TaskState and DagState to _actually_ be an enum, and have to go rewrite a large chunk of core code and potentially break compatability with many providers since it'll change the values of all the states to be of type `EnumValue`.
Would it help to initially make them a [`StrEnum`](https://docs.python.org/3/library/enum.html#others), and slowly convert the type hints to take an enum value instead of str?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819243475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andrewgodwin commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
andrewgodwin commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-827893328
> No interest in calling it `TaskExecutionState` instead of `TaskState`? 🙂
I'd rather we call it `TaskInstanceState` honestly, but right now I'm just trying to see if this will even work type-wise :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819648630
> I think the "stateful" tasks doesn't conflict with this -- Airflow essentially has a state-machine (if a very poorly defined one) for what states a TI/DR can be in, so adding a whole new state is more work than just changing the value in the DB column, so by making this an explicit enum class it would hopefully make it clearer that you can't make up your own values to store in ti.state and expect it to Just Work.
I don't mean that the _two concepts_ conflict just that I would like to reserve the _name_ `TaskState` and `DagState`, potentially, for use with "stateful" tasks.
just trying to think if there's a name we could use for the _execution_ states enum which would not clobber that.
but, maybe owing to the fact that there already is an "execution state" concept, if "stateful tasks" is ever implememted it might be prudent to use the name "watermark" or "artifact" instead.
But I think potentially TaskState could be a good name for a stateful tasks framework, and if we called this enum something like TaskRunState or TaskExecutionState there would be no conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819648630
> I think the "stateful" tasks doesn't conflict with this -- Airflow essentially has a state-machine (if a very poorly defined one) for what states a TI/DR can be in, so adding a whole new state is more work than just changing the value in the DB column, so by making this an explicit enum class it would hopefully make it clearer that you can't make up your own values to store in ti.state and expect it to Just Work.
I don't mean that the _two concepts_ conflict just that I would like to reserve the _name_ `TaskState` and `DagState`, potentially, for use with "stateful" tasks.
just trying to think if there's a name we could use for the _execution_ states enum which would not clobber that.
but, maybe owing to the fact that there already is an "execution state" concept, if "stateful tasks" is ever implememted it might be prudent to use the name "watermark" or "artifact" instead.
But I think potentially TaskState could be a good name for a stateful tasks framework, and if we called this enum something like TaskRunState or TaskExecutionState -- or perhaps more accurately TaskInstanceExecutionState there would be no conflict.
Another idea would be to nest the different execution states under a States class like States.dag_run.RUNNING etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#discussion_r631831440
##########
File path: airflow/utils/state.py
##########
@@ -16,70 +16,110 @@
# specific language governing permissions and limitations
# under the License.
+from enum import Enum
+from typing import Dict, FrozenSet, Tuple
+
from airflow.settings import STATE_COLORS
+from airflow.utils.types import Optional
-class State:
+class TaskState(str, Enum):
"""
- Static class with task instance states constants and color method to
- avoid hardcoding.
+ Enum that represents all possible states that a Task Instance can be in.
+
+ Note that None is also allowed, so always use this in a type hint with Optional.
"""
- # scheduler
- NONE = None # type: None
- REMOVED = "removed"
- SCHEDULED = "scheduled"
+ # Set by the scheduler
+ # None - Task is created but should not run yet
+ REMOVED = "removed" # Task vanished from DAG before it ran
+ SCHEDULED = "scheduled" # Task should run and will be handed to executor soon
+
+ # Set by the task instance itself
+ QUEUED = "queued" # Executor has enqueued the task
+ RUNNING = "running" # Task is executing
+ SUCCESS = "success" # Task completed
+ SHUTDOWN = "shutdown" # External request to shut down
+ FAILED = "failed" # Task errored out
+ UP_FOR_RETRY = "up_for_retry" # Task failed but has retries left
+ UP_FOR_RESCHEDULE = "up_for_reschedule" # A waiting `reschedule` sensor
+ UPSTREAM_FAILED = "upstream_failed" # One or more upstream deps failed
+ SKIPPED = "skipped" # Skipped by branching or some other mechanism
+ SENSING = "sensing" # Smart sensor offloaded to the sensor DAG
+
+
+class DagState(str, Enum):
+ """
+ Enum that represents all possible states that a DagRun can be in.
- # set by the executor (t.b.d.)
- # LAUNCHED = "launched"
+ These are "shared" with TaskState in some parts of the code, so make
+ sure they don't drift.
+ """
- # set by a task
- QUEUED = "queued"
RUNNING = "running"
SUCCESS = "success"
- SHUTDOWN = "shutdown" # External request to shut down
FAILED = "failed"
- UP_FOR_RETRY = "up_for_retry"
- UP_FOR_RESCHEDULE = "up_for_reschedule"
- UPSTREAM_FAILED = "upstream_failed"
- SKIPPED = "skipped"
- SENSING = "sensing"
-
- task_states = (
- SUCCESS,
- RUNNING,
- FAILED,
- UPSTREAM_FAILED,
- SKIPPED,
- UP_FOR_RETRY,
- UP_FOR_RESCHEDULE,
- QUEUED,
- NONE,
- SCHEDULED,
- SENSING,
- REMOVED,
+
+
+class State:
+ """
+ Static class with task instance states constants and color method to
+ avoid hardcoding.
+ """
+
+ # Backwards-compat constants for code that does not yet use the enum
+ # These first three are shared by DagState and TaskState
+ SUCCESS = TaskState.SUCCESS
+ RUNNING = TaskState.RUNNING
+ FAILED = TaskState.FAILED
+
+ # These are TaskState only
+ NONE = None
+ REMOVED = TaskState.REMOVED
+ SCHEDULED = TaskState.SCHEDULED
+ QUEUED = TaskState.QUEUED
+ SHUTDOWN = TaskState.SHUTDOWN
+ UP_FOR_RETRY = TaskState.UP_FOR_RETRY
+ UP_FOR_RESCHEDULE = TaskState.UP_FOR_RESCHEDULE
+ UPSTREAM_FAILED = TaskState.UPSTREAM_FAILED
+ SKIPPED = TaskState.SKIPPED
+ SENSING = TaskState.SENSING
+
+ task_states: Tuple[Optional[TaskState], ...] = (
+ None,
+ TaskState.SUCCESS,
+ TaskState.RUNNING,
+ TaskState.FAILED,
+ TaskState.UPSTREAM_FAILED,
+ TaskState.SKIPPED,
+ TaskState.UP_FOR_RETRY,
+ TaskState.UP_FOR_RESCHEDULE,
+ TaskState.QUEUED,
+ TaskState.SCHEDULED,
+ TaskState.SENSING,
+ TaskState.REMOVED,
Review comment:
```suggestion
task_states: Tuple[Optional[TaskState], ...] = (None,) + tuple(TaskState)
```
That way it can't ever get out of sync https://docs.python.org/3/library/enum.html#iteration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819243475
it's possible to have a stringy enum class, did you explore that?
see here https://stackoverflow.com/a/63028809/2761682
one concern i have with this PR is the creation of a TaskState class.
there has been periodic discussion on mailing list re adding support for stateful tasks and of late some renewed interest in the topic. adding DagState and TaskState classes would potentially be in conflict with that. if we do create stateful tasks, i think that TaskState as the model through which we store state (i.e. an artifact representin the task's state) would be a better usage of the word than an enum representing the execution state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andrewgodwin commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
andrewgodwin commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-827734822
Not at all - I was not familiar with the ability to add arbitrary type bases to `Enum`! I'll try that, it should be significantly better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-858737020
@andrewgodwin ping -- when you have a few spare cycles would be good to revisit this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andrewgodwin commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
andrewgodwin commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-827903722
I get your point, but we also can't escape that it is the possible values of the `state` column on `TaskInstance`. I agree the naming that's there already isn't _amazing_, but there's also something to be said for the principle of least surprise :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #15285:
URL: https://github.com/apache/airflow/pull/15285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819648630
> I think the "stateful" tasks doesn't conflict with this -- Airflow essentially has a state-machine (if a very poorly defined one) for what states a TI/DR can be in, so adding a whole new state is more work than just changing the value in the DB column, so by making this an explicit enum class it would hopefully make it clearer that you can't make up your own values to store in ti.state and expect it to Just Work.
I don't mean that the _two concepts_ conflict just that I would like to reserve the _name_ `TaskState` and `DagState`, potentially, for use with "stateful" tasks.
just trying to think if there's a name we could use for the _execution_ states enum which would not clobber that.
but, maybe owing to the fact that there already is an "execution state" concept, if "stateful tasks" is ever implememted it might be prudent to use the name "watermark" or "artifact" instead.
But I think potentially TaskState could be a good name for a stateful tasks framework, and if we called this enum something like TaskRunState or TaskExecutionState -- or perhaps more accurately TaskInstanceExecutionState there would be no conflict.
Another idea would be to nest the different execution state enums under a States class like States.dag_run.RUNNING etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#discussion_r661091782
##########
File path: airflow/utils/state.py
##########
@@ -96,17 +127,17 @@ def color_fg(cls, state):
return 'white'
return 'black'
- running = frozenset([RUNNING, SENSING])
+ running: FrozenSet[TaskInstanceState] = frozenset([TaskInstanceState.RUNNING, TaskInstanceState.SENSING])
Review comment:
Huh, surprised this type hint is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-875460102
D'oh, we had two changes in flight -- this one and @ephraimbuddy's https://github.com/apache/airflow/pull/16401, so we'll need to add Queued state to DagRuns too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819243475
it's possible to have a stringy enum class, did you explore that?
see here https://stackoverflow.com/a/63028809/2761682
one concern i have with this PR is the creation of a TaskState class.
there has been periodic discussion on mailing list re adding support for stateful tasks and of late some renewed interest in the topic. adding DagState and TaskState classes would potentially be in conflict with that. if we do create stateful tasks, i think that TaskState as the model through which we store state (i.e. an artifact representin the task's state) would be a better usage of the word than an enum representing the execution state.
i should add, it would probably be more accurate to describe these as TaskInstanceState and DagRunState -- the task and dag themselves don't actually have states of any kind at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-874898207
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andrewgodwin commented on a change in pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
andrewgodwin commented on a change in pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#discussion_r661607845
##########
File path: airflow/utils/state.py
##########
@@ -96,17 +127,17 @@ def color_fg(cls, state):
return 'white'
return 'black'
- running = frozenset([RUNNING, SENSING])
+ running: FrozenSet[TaskInstanceState] = frozenset([TaskInstanceState.RUNNING, TaskInstanceState.SENSING])
Review comment:
It may not be, I just like adding explicit type hints where I think it helps the reader of the code understand what's going on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819429445
> there has been periodic discussion on mailing list re adding support for stateful tasks and of late some renewed interest in the topic. adding DagState and TaskState classes would potentially be in conflict with that. if we do create stateful tasks, i think that TaskState as the model through which we store state (i.e. an artifact representin the task's state) would be a better usage of the word than an enum representing the execution state.
I think the "stateful" tasks doesn't conflict with this -- Airflow essentially has a state-machine (if a _very_ poorly defined one) for what states a TI/DR can be in, so adding a whole new state is more work than just changing the value in the DB column, so by making this an explicit enum class it would hopefully make it clearer that you can't make up your own values to store in ti.state and expect it to Just Work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819648630
> I think the "stateful" tasks doesn't conflict with this -- Airflow essentially has a state-machine (if a very poorly defined one) for what states a TI/DR can be in, so adding a whole new state is more work than just changing the value in the DB column, so by making this an explicit enum class it would hopefully make it clearer that you can't make up your own values to store in ti.state and expect it to Just Work.
I don't mean that the _two concepts_ conflict just that I would like to reserve the _name_ `TaskState` and `DagState`, potentially, for use with "stateful" tasks.
just trying to think if there's a name we could use for the _execution_ states enum which would not clobber that.
but, maybe owing to the fact that there already is an "execution state" concept, if "stateful tasks" is ever implememted it might be prudent to use the name "watermark" or "artifact" instead.
But I think potentially TaskState could be a good name for a stateful tasks framework, and if we called this enum something like TaskRunState or TaskExecutionState -- or perhaps more accurately TaskInstanceExecutionState there would be no conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-827892600
No interest in calling it `TaskExecutionState` instead of `TaskState`? 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-862166289
Code looks good now -- @dstandish Are you happy with these names (`TaskState` and `DagState`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andrewgodwin commented on a change in pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
andrewgodwin commented on a change in pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#discussion_r660776256
##########
File path: airflow/utils/state.py
##########
@@ -16,70 +16,100 @@
# specific language governing permissions and limitations
# under the License.
+from enum import Enum
+from typing import Dict, FrozenSet, Tuple
+
from airflow.settings import STATE_COLORS
+from airflow.utils.types import Optional
-class State:
+class TaskState(str, Enum):
"""
- Static class with task instance states constants and color method to
- avoid hardcoding.
+ Enum that represents all possible states that a Task Instance can be in.
+
+ Note that None is also allowed, so always use this in a type hint with Optional.
"""
- # scheduler
- NONE = None # type: None
- REMOVED = "removed"
- SCHEDULED = "scheduled"
+ # Set by the scheduler
+ # None - Task is created but should not run yet
+ REMOVED = "removed" # Task vanished from DAG before it ran
+ SCHEDULED = "scheduled" # Task should run and will be handed to executor soon
- # set by the executor (t.b.d.)
- # LAUNCHED = "launched"
+ # Set by the task instance itself
+ QUEUED = "queued" # Executor has enqueued the task
+ RUNNING = "running" # Task is executing
+ SUCCESS = "success" # Task completed
+ SHUTDOWN = "shutdown" # External request to shut down
+ FAILED = "failed" # Task errored out
+ UP_FOR_RETRY = "up_for_retry" # Task failed but has retries left
+ UP_FOR_RESCHEDULE = "up_for_reschedule" # A waiting `reschedule` sensor
+ UPSTREAM_FAILED = "upstream_failed" # One or more upstream deps failed
+ SKIPPED = "skipped" # Skipped by branching or some other mechanism
+ SENSING = "sensing" # Smart sensor offloaded to the sensor DAG
+
+ def __str__(self) -> str: # pylint: disable=invalid-str-returned
+ return self.value
+
+
+class DagState(str, Enum):
+ """
+ Enum that represents all possible states that a DagRun can be in.
+
+ These are "shared" with TaskState in some parts of the code, so make
+ sure they don't drift.
Review comment:
I will make the docstring more direct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819648630
> I think the "stateful" tasks doesn't conflict with this -- Airflow essentially has a state-machine (if a very poorly defined one) for what states a TI/DR can be in, so adding a whole new state is more work than just changing the value in the DB column, so by making this an explicit enum class it would hopefully make it clearer that you can't make up your own values to store in ti.state and expect it to Just Work.
I don't mean that the _two concepts_ conflict just that I would like to reserve the _name_ `TaskState` and `DagState`, potentially, for use with "stateful" tasks.
just trying to think if there's a name we could use for the _execution_ states enum which would not clobber that.
but, maybe owing to the fact that there already is an "execution state" concept, if "stateful tasks" is ever implememted it might be prudent to use the name "watermark" or "artifact" instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-827900483
> I'm just trying to see if this will even work type-wise
Totally makes sense :)
> I'd rather we call it TaskInstanceState honestly
Let me offer an argument for _not_ this.
It's possible to use xcom as a store of task instance state, because it is at the grain of task instance (task + execution date).
So we can imagine that XCom could have reasonably been called TaskInstanceState.
What this enum represents is something very different. It's the TaskInstance's _execution_ state -- not a way of storing state data for a task instance.
That's why I like the idea of adding the word `execution` or `run`.
Separately, I think we could justify going without the word _instance_ because this enum represents possible values for task execution states in the abstract, and it is maybe irrelevant that the when a task gets combined with an execution date it is called a task instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#discussion_r661701608
##########
File path: airflow/utils/state.py
##########
@@ -96,17 +127,17 @@ def color_fg(cls, state):
return 'white'
return 'black'
- running = frozenset([RUNNING, SENSING])
+ running: FrozenSet[TaskInstanceState] = frozenset([TaskInstanceState.RUNNING, TaskInstanceState.SENSING])
Review comment:
Yep. I also often do that even if not technically needed to make it explicit. I think code should be much more optimised for reader's time than writer's time to write it ;).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andrewgodwin commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
andrewgodwin commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-870750500
I decided to rename the classes under the banner of being more explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
dstandish commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-827900483
> I'm just trying to see if this will even work type-wise
Totally makes sense :)
> I'd rather we call it TaskInstanceState honestly
Let me argue for _not_ this.
It's possible to use xcom as a store of task instance state, because it is at the grain of task instance (task + execution date).
So we can imagine that XCom could have reasonably been called TaskInstanceState.
What this enum represents is something very different. It's the TaskInstance's _execution_ state -- not a way of storing state data for a task instance.
That's why I like the idea of adding the word `execution` or `run`.
Separately, I think we could justify going without the word _instance_ because this enum represents possible values for task execution states in the abstract, and it is maybe irrelevant that the when a task gets combined with an execution date it is called a task instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#discussion_r631828382
##########
File path: airflow/utils/state.py
##########
@@ -16,70 +16,110 @@
# specific language governing permissions and limitations
# under the License.
+from enum import Enum
+from typing import Dict, FrozenSet, Tuple
+
from airflow.settings import STATE_COLORS
+from airflow.utils.types import Optional
-class State:
+class TaskState(str, Enum):
"""
- Static class with task instance states constants and color method to
- avoid hardcoding.
+ Enum that represents all possible states that a Task Instance can be in.
+
+ Note that None is also allowed, so always use this in a type hint with Optional.
"""
- # scheduler
- NONE = None # type: None
- REMOVED = "removed"
- SCHEDULED = "scheduled"
+ # Set by the scheduler
+ # None - Task is created but should not run yet
+ REMOVED = "removed" # Task vanished from DAG before it ran
+ SCHEDULED = "scheduled" # Task should run and will be handed to executor soon
+
+ # Set by the task instance itself
+ QUEUED = "queued" # Executor has enqueued the task
+ RUNNING = "running" # Task is executing
+ SUCCESS = "success" # Task completed
+ SHUTDOWN = "shutdown" # External request to shut down
+ FAILED = "failed" # Task errored out
+ UP_FOR_RETRY = "up_for_retry" # Task failed but has retries left
+ UP_FOR_RESCHEDULE = "up_for_reschedule" # A waiting `reschedule` sensor
+ UPSTREAM_FAILED = "upstream_failed" # One or more upstream deps failed
+ SKIPPED = "skipped" # Skipped by branching or some other mechanism
+ SENSING = "sensing" # Smart sensor offloaded to the sensor DAG
+
Review comment:
```suggestion
def __str__(self) -> str: # pylint: disable=invalid-str-returned
return self.value
```
Is what we did in DagRunType
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819424297
> if you want to make this an enum, why can't you just keep the single State class and make it enum, or enum-like?
I think that’d take away most of the reason of having an enum class. The main advantage of enums is to prevent invalid values from being passed (they’d result in a typing or runtime error). Combining all the possible state values into one enum means not all state values make sense for all contexts (some states are impossible on task instances, for example), and kind of defeats the purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andrewgodwin commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
andrewgodwin commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-827866899
I've reworked this to use a str-type Enum, but the enum values are still odd - calling `str()` on them outputs the name of the value as `TaskState.FAILED`, not just `failed`. I don't think the code is going to like it.
I suspect it's going to massively blow up the test suite, but I want to push it up and run here just to be sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #15285: Fixed #9387: Add State types for tasks and DAGs
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #15285:
URL: https://github.com/apache/airflow/pull/15285#issuecomment-819428250
> > if you want to make this an enum, why can't you just keep the single State class and make it enum, or enum-like?
>
> I think that’d take away most of the reason of having an enum class. The main advantage of enums is to prevent invalid values from being passed (they’d result in a typing or runtime error). Combining all the possible state values into one enum means not all state values make sense for all contexts (some states are impossible on task instances, for example), and kind of defeats the purpose.
Exactly this -- there are some states that only apply to TaskInstances, but that don't apply to DagRuns or (Base)Job classes -- "upstream_failed" for instance (there are more).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org