You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/14 16:25:38 UTC
[GitHub] [airflow] MatrixManAtYrService opened a new issue, #23020: Names for expanded tasks
MatrixManAtYrService opened a new issue, #23020:
URL: https://github.com/apache/airflow/issues/23020
### Description
Airflow currently exposes `map_index` to the user as a way of distinguishing between tasks in an expansion. The index is unlikely to be meaningful to the user. They probably have their own label for this action. I'm requesting that we allow them to add that label.
To see the problem, consider a dag that sends email to a list of users which is generated at runtime:
```python3
with DAG(...) as dag:
@dag.task
def get_account_status():
return [
{
"NAME": "Wintermute",
"EMAIL": "wintermute@tessier-ashpool.com",
"STATUS": "active",
},
{
"NAME": "Hojo",
"EMAIL": "ops@research.shinra.com",
"STATUS": "delinquent",
},
]
BashOperator.partial(
task_id="send_email",
bash_command=dedent(
"""
cat <<- EOF | tee | mailx -s "your account" $EMAIL
Dear $NAME,
Your account status is $STATUS.
EOF
"""
),
).expand(env=get_account_status())
```
Notice that in the grid view, it's not obvious which task goes with which user:
<img width="1106" alt="Screen Shot 2022-04-14 at 8 56 09 AM" src="https://user-images.githubusercontent.com/5834582/163418431-3180a29d-b9c0-4bbc-9a80-05ad5e4f34e7.png">
### Use case/motivation
I'd like to be able to explicitly assign a name to each expanded task, that way I can later go look at the right one. I would like this name to be used (when available) anywhere that the user interacts with the expanded task.
I'm not sure if we should replace `map_index` with `mapped_task_key`, or just add a name as a separate thing. The replacement sounds more invasive, but it would allow us to generate better names in cases where more than one kwarg is mapped to.
For instance, this expansion generates four instances.
```python3
BashOperator.partial(task_id="greet").expand(
bash_command=["echo hello $USER", "echo goodbye $USER"],
env=[{"USER": "foo"}, {"USER": "bar"}],
)
```
If the user doesn't supply names (via whatever API we come up with), do we want to call them `1` `2` `3` and `4`? Or should they be more descriptive like
- `bash_command_1_env_1`
- `bash_command_1_env_2`
- `bash_command_2_env_1`
- `bash_command_2_env_2`
I don't know. I'm creating this issue so we have a place to discuss it.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1204662843
It just occurred to me that this is essentially a part of #22073. What we (users) actually want is a more customisable way to identify things (in this instance, a mapped task _instance_), and if we look past the assumption that a mapped task instance is “task_id + map_index”, we simply need a better way for the user to tell “what is this thing” in the Airflow UI. So let’s keep track of that issue instead to make sure whatever solution we come up for it correctly considers map_index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1099391966
When I think about how the API might look, I come up with this:
```python
BashOperator.partial(task_id="say").expand_dicts(
bash_command={
"generic": "echo hello",
"cow": "cowsay $MESSAGE",
"pikachu": "pokemonsay -p Pikachu $MESSAGE",
},
env={
"pikachu": {"MESSAGE": "pika"},
"cow": {"MESSAGE": "moo"}
},
)
```
This would create three tasks, and the "generic" one wouldn't define any env vars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chrisluedtke commented on issue #23020: Names for expanded tasks
Posted by "chrisluedtke (via GitHub)" <gi...@apache.org>.
chrisluedtke commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1695930725
Can this issue please be re-opened? The implementation of PR #32520 to deliver #22073 does not enable users to define unique id's (map index) for mapped task instances.
Maybe the desired behavior could be added to `.expand()` as a reserved `map_index` kwarg?
```python
import pendulum
from airflow.decorators import dag, task
@dag(
schedule=None,
start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
catchup=False,
tags=["example"],
)
def demo():
@task
def handle_animal(animal: dict):
print("Run away")
animals = [{'id': 'lion'}, {'id': 'tiger'}, {'id': 'bear'}]
handle_animal.expand(animal=animals, map_index=[a['id'] for a in animals])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1099398215
+1 I'm super happy to do this in the UI as soon as it can access that data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "damjad (via GitHub)" <gi...@apache.org>.
damjad commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1932675131
@RNHTTR are you working on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1204339810
BTW. I hate MySQL and how it limits us in design, and I proposed already few times to remove it, but we came to the conclusion that supporting the community running MySQL installation historically is more important. This is unfortunately one of the limitations it introduces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr closed issue #23020: Names for expanded tasks
URL: https://github.com/apache/airflow/issues/23020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chrisluedtke commented on issue #23020: Names for expanded tasks
Posted by "chrisluedtke (via GitHub)" <gi...@apache.org>.
chrisluedtke commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1714092915
> Reopening issue like that does not change much. Someone willl have to implement it.
Yes, the purpose of an issue is not to implement a feature. The purpose of an issue is to gather interest and discussion around a topic and to inspire implementation.
> We are not "Issue-driven" we are "code driven" - so PRs are more important than issues, especially on new (small-ish) features like that.
The importance of PRs versus issues is not relevant. The original feature request is valid and should be kept open until the feature is delivered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "paulg-mscience (via GitHub)" <gi...@apache.org>.
paulg-mscience commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1861262894
@kaxil Is this something you can influence and push through?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1204329535
I think you can forget about this.
You've just hit reality train.
Look there: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-42+Dynamic+Task+Mapping - see this part:
"Rather than overloading the task_id argument to `airflow tasks run` (i.e. having a task_id of `run_after_loop[0]`) we will add a new `--mapping-id` argument to `airflow tasks run` -- this value will be a ~JSON-encoded~ an integer specifying the index/position of the mapping."
We have to support MySQL and the problem with MySQL is that index size is limited. VERY limited. Depending on the type of encooding it might be even 720 characters. And task-id + dag_id + task_index already exceed the limit. And there is no way around it - and this was the main reason (I believe) we had to use integer, even if originally we planned not even a name but JSON-encoded list of parameters( which was far better for uniqueness - because it was automated).
But this is just what I saw - by observing it being implemented, so I might be wrong on that account
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Gollum999 commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
Gollum999 commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1204302199
For the sake of discussion, how do people feel about letting these names entirely *replace* `map_index`?
My primary motivation is that an integer-based `map_index` can get weird if the list of tasks changes when re-running a DAG. In my experience, you can end up with missing and/or duplicate tasks due to the mismatched indices. As a result you are generally forced to re-run all of the mapped task instances (if not the entire DAG Run), even if 99% of the tasks are unchanged. I can give more concrete examples if anyone is interested.
I'd propose letting the mapped args become (part of?) the primary key for the mapped task. This would solve OP's goal of improving the UX of mapped tasks (no more abstract indices), but would also allow for some amount of consistency between runs (not just for re-runs, but potentially between separate DAG Runs as well).
Conceptually I think this is doable since the args already must be JSON serializable, but I'm sure the implementation would be more complex than I am imagining.
A couple considerations that come to mind:
1. These IDs ultimately need to be unique. So something like `map_index` might still be needed to augment the key and resolve duplicates. I imagine this could be similar to how duplicate task_ids are resolved (`task`, `task__1`, etc.).
2. The values used to generate the list of dynamic tasks could be very large and complex. So it seems like you would need the ability to override the generated name with a "short name" when args are too complex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1880089536
I'm interested in taking this on (or at least moving it forward). Would it be sufficient to start by:
1. Adding something like `named_mapped_index` to BaseOperator and as a column in the `TaskInstance` model
2. Rendering the named index in the UI in the grid view and TaskInstance view
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1880105381
> I'm interested in taking this on (or at least moving it forward). Would it be sufficient to start by:
>
> 1. Adding something like `named_mapped_index` to BaseOperator and as a column in the `TaskInstance` model
> 2. Rendering the named index in the UI in the grid view and TaskInstance view
Yes. that's a good start.
@uranusjr can help guide you Ryan if you need any help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1100307134
Exposing a function might not be a good idea, since the function needs to be executed by the scheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "pykenny (via GitHub)" <gi...@apache.org>.
pykenny commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-2646055526
Not trying to reopen the issue (since feature to set displayed name has been shipped). I think the idea of letting users to define their own set of index ID not only hits functional limit on MySQL, but also can lead to the problem similar to this report:
- https://github.com/apache/airflow/issues/24701
Users can unintentionally generate duplicated IDs during the mapping process, and both overwriting duplicated one (last-write-wins) or appending suffix (like how task IDs are determined) may not fit user expectation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "paulg-mscience (via GitHub)" <gi...@apache.org>.
paulg-mscience commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1861257150
It looks like this issue has lost moment but we really need some sort of tag or label attribute that could be assigned per mapped task instance and surfaced in the airflow UI to aide usability, trouble-shooting and monitoring. This tag/label would be in addition to the numeric index and not a replacement for the index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1933020783
> @RNHTTR are you working on it?
yep! https://github.com/apache/airflow/pull/36797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1204652488
Even without the MySQL limitation, we still need some sort of index to keep track of the ordering, otherwise there’s no way to know which upstream value maps to which worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1100751912
It’s possible, but I think separating the logic into task would be much more complicated to implement, since we don’t really have a good way to communiate arbitrary information between tasks (except XCom which is probably not the right place for this). Doing it as a part of the task run as Jarek suggested is better. We already have pre-execute hooks, so it can fit around there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23020: Names for expanded tasks
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1714575810
@chrisluedtke . sure no problem. It does not change a thing to open an issue that has not been discussed for a year. But let me reopen and assign to you, hopefully you can lead it to completion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1204867427
> It just occurred to me that this is essentially a part of #22073. What we (users) actually want is a more customisable way to identify things (in this instance, a mapped task _instance_), and if we look past the assumption that a mapped task instance is “task_id + map_index”, we simply need a better way for the user to tell “what is this thing” in the Airflow UI. So let’s keep track of that issue instead to make sure whatever solution we come up for it correctly considers map_index.
Fully agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1100149376
re: `zip`, I was thinking that this would make that easier. If you want to zip together two lists of length _n_ then you can just use a dict comprehension that gives each item one of _n_ keys--items with the same key get zipped together. I guess it wouldn't really extend to stuff like `accumulate` though.
I didn't include user-defined functions in my proposal because I was worried that:
- If something goes wrong in that function, how do you communicate to them which one had the problem?
- User has to wonder what airflow will do if they return the same key twice--with dicts the collision is handled when the user is building the object.
- Other scary runtime concerns?
It's good to know that maybe my third concern isn't so scary after all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1100470713
Could it go in a separate task that goes upstream of the expansion? That gives us a way to communicate errors back to the user.
I'm imagining such structures appearing in the UI as task groups that you can't expand until they've run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1861821868
> It looks like this issue has lost moment but we really need some sort of tag or label attribute that could be assigned per mapped task instance and surfaced in the airflow UI to aide usability, trouble-shooting and monitoring. This tag/label would be in addition to the numeric index and not a replacement for the index.
No. It did not loose moment. Simply no-one picked it up. As mentioned above - the best thing to do when you care about something to be implemented is to ... implement it. Second best is to find someone who can implement it. Possibly sponsor somoene's work. I thought I explained it above that reopening an issue does not **magically** make it implemented. You need someone who will put their time and effort into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23020: Names for expanded tasks
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1696503814
> Can this issue please be re-opened? The implementation of PR https://github.com/apache/airflow/pull/32520 to deliver https://github.com/apache/airflow/issues/22073 does not enable users to define unique id's (map index) for mapped task instances.
Reopening issue like that does not change much. Someone willl have to. implement it. If you have idea how to implement, you can definitely propose PR. You can also make suggestions in the PR opened. We are not "Issue-driven" we are "code driven" - so PRs are more important than issues, especially on new (small-ish) features like that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1100018655
3. How about (on top of auto-generated id) add a function to return the task_id(). If we calculate it eagerly at task execution time, there are no security concerns. and it will be far more versatile than template-based id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1100344089
I thought it might be run at execution time and update task instance then. It won't work for queued state but maybe that's not really needed.
If we allow Jinja template (one of the options) to be run in scheduler it's also a bit dangerous
Clear demarcation that you update it at execution time should be fine. This is what for example GitHub Actions do. They use the expression language and display the expression until the task start to be executed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Names for expanded tasks [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1965800647
Implemented in #36797.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1099997002
We could use a heuristic similar to pytest. I’m not sure what their exact logic is, but it seems to be something like if the mapped value is simple (int, float, or a short-ish str), use `str()` to convert it directly, otherwise use something like `arg1` (we could use `[0]`, `[1]` etc.).
Some things I can think of now:
1. Calculating the label may be expensive since this requires the view to re-pull upstream XCom. We may need to do this eagerly when the task instance is being executed, which means adding a new field on TaskInstance to store this for later use.
2. The logic becomes difficult when we eventually add functional tools to task mapping e.g. `produce`, `zip` etc.
3. Pytest allows using `id` to supply custom IDs when the auto-generated label is not good enough (but you don’t want the non-descriptive `arg1`). We can’t do this since the indexes are dynamically calculated, but maybe an argument on BaseOperator to let the user provide a string template for this would be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1108495026
@MatrixManAtYrService - maybe you want to take a stab at it and implement it :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1204869545
Closing this one as duplicate of #22073 (and I make a comment on that issue to make sure it includes mapped tasks).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #23020: Names for expanded tasks
URL: https://github.com/apache/airflow/issues/23020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on issue #23020: Names for expanded tasks
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on issue #23020:
URL: https://github.com/apache/airflow/issues/23020#issuecomment-1111431301
I'd like to, but first I'll have to argue with a certain kanban board. I'll report back with a link to a draft PR if successful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org