You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ayush3singh (via GitHub)" <gi...@apache.org> on 2025/05/08 22:13:12 UTC
[PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
ayush3singh opened a new pull request, #50376:
URL: https://github.com/apache/airflow/pull/50376
Fixes #50330 ###
This PR adds `log_url` to
`RuntimeTaskInstance`
`ti.log_url`
`Context`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2081096254
##########
airflow-core/src/airflow/executors/workloads.py:
##########
@@ -71,6 +71,8 @@ class TaskInstance(BaseModel):
context_carrier: dict | None = None
queued_dttm: datetime | None = None
+ log_url: str | None = None
Review Comment:
Basically we have all the information in task_runner.py to construct the uri, lets do it there, look in `get_template_context`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "ayush3singh (via GitHub)" <gi...@apache.org>.
ayush3singh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2083271436
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: ti.log_url if getattr(ti, "log_url", -1) != -1 else None,
Review Comment:
Hi @amoghrajesh
without this log_url gets included in the final object `TaskInstanceInfo` and although it is set to none we have to adjust the utils testcases to add `log_url` in assert statements as shown below
`
assert dict(TaskInstanceInfo(runtime_ti)) == {
"log_url": None,
`
and since log_url is now expected but its not available in Airflow 3.0.0 I guess, so below compatibility test fails:
[provider distributions tests / Compat 3.0.0:P3.9](https://github.com/apache/airflow/actions/runs/14947763348/job/41993407704#logs)
if we make this change then the Test passes as it will keep a place holder if not available.
I'm fairly new in contribution to airflow repo, please let me know if I'm missing out on something.
Thanks for review comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "ayush3singh (via GitHub)" <gi...@apache.org>.
ayush3singh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2084054293
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: getattr(ti, "log_url", None),
Review Comment:
Hi @kacpermuda
without this , the below compatibility test fails:
provider distributions tests / Compat 3.0.0:P3.9
if we make this change then the Test succeeds as it will keep a place holder if not available in object ( in case of Airflow)
Keeping in `includes` list will add attribute only if original json hasvthe key, but in Airflow 3.0 its not present
Thanks for review comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "ayush3singh (via GitHub)" <gi...@apache.org>.
ayush3singh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2081262718
##########
airflow-core/src/airflow/executors/workloads.py:
##########
@@ -71,6 +71,8 @@ class TaskInstance(BaseModel):
context_carrier: dict | None = None
queued_dttm: datetime | None = None
+ log_url: str | None = None
Review Comment:
I agree @amoghrajesh
Thanks for the advise, I have added made changes to read everything on sdk instead of passing it from server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2081061600
##########
airflow-core/src/airflow/executors/workloads.py:
##########
@@ -71,6 +71,8 @@ class TaskInstance(BaseModel):
context_carrier: dict | None = None
queued_dttm: datetime | None = None
+ log_url: str | None = None
Review Comment:
Instead of passing it around, can we construct it on the supervisor side itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2083107941
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: ti.log_url if getattr(ti, "log_url", -1) != -1 else None,
Review Comment:
Why do we need this one?
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -158,6 +183,9 @@ def get_template_context(self) -> Context:
validated_params = process_params(self.task.dag, self.task, dag_run_conf, suppress_exception=False)
+ self.log_url = self.get_log_url_from_ti()
+ self.mark_success_url = self.get_mark_success_url()
Review Comment:
Can we move this logic to "startup" instead? It doens't necessarily need to have context in order to set these.
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -137,6 +137,10 @@ class RuntimeTaskInstance(TaskInstance):
rendered_map_index: str | None = None
+ log_url: str | None = None
+
+ mark_success_url: str | None = None
Review Comment:
Also i do not understand the use case/ motivation behind this one
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -137,6 +137,10 @@ class RuntimeTaskInstance(TaskInstance):
rendered_map_index: str | None = None
+ log_url: str | None = None
+
+ mark_success_url: str | None = None
Review Comment:
Let's please extract this one to a separate PR.
##########
task-sdk/tests/task_sdk/execution_time/test_task_runner.py:
##########
@@ -1028,6 +1028,8 @@ def test_get_context_without_ti_context_from_server(self, mocked_parse, make_ti_
_ti_context_from_server=None,
start_date=start_date,
)
+ runtime_ti.log_url = runtime_ti.get_log_url_from_ti()
Review Comment:
Can we add a test that gets that asserts the log_url?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "messense (via GitHub)" <gi...@apache.org>.
messense commented on PR #50376:
URL: https://github.com/apache/airflow/pull/50376#issuecomment-2921580811
I'm getting `Failed to send notification: 'airflow.sdk.execution_time.task_runner.RuntimeTaskInstance object' has no attribute 'mark_success_url'` even with this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2083996702
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: getattr(ti, "log_url", None),
Review Comment:
I'm not sure if it needs to be here. It's already present in `includes` above, so it should work exactly like `getattr(ti, "log_url", None)`. Can you remove it and see?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #50376:
URL: https://github.com/apache/airflow/pull/50376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #50376:
URL: https://github.com/apache/airflow/pull/50376#issuecomment-2921617812
CC @sunank200 do you want to take a stab at it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "ayush3singh (via GitHub)" <gi...@apache.org>.
ayush3singh commented on PR #50376:
URL: https://github.com/apache/airflow/pull/50376#issuecomment-2864552914
@tirkarthi for review and guidence
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2083996702
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: getattr(ti, "log_url", None),
Review Comment:
I'm not sure if it needs to be here. It's already present in `includes` above, so it should work exactly like `getattr(ti, "log_url", None)`. Can you remove it ant see?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "ayush3singh (via GitHub)" <gi...@apache.org>.
ayush3singh commented on PR #50376:
URL: https://github.com/apache/airflow/pull/50376#issuecomment-2921722997
@amoghrajesh
I can patch it since we removed the change as it was out of scope.
Is it fine if I raise a pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "ayush3singh (via GitHub)" <gi...@apache.org>.
ayush3singh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2083590789
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: ti.log_url if getattr(ti, "log_url", -1) != -1 else None,
Review Comment:
Thanks @mobuchowski
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "ayush3singh (via GitHub)" <gi...@apache.org>.
ayush3singh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2084054293
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: getattr(ti, "log_url", None),
Review Comment:
Hi @kacpermuda
without this , the below compatibility test fails:
provider distributions tests / Compat 3.0.0:P3.9
if we make this change then the Test succeeds as it will keep a place holder if not available in object ( in case of Airflow3.0.0)
Keeping in `includes` list will add attribute only if original json has the key, but in Airflow 3.0 its not present
Thanks for review comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2083385278
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: ti.log_url if getattr(ti, "log_url", -1) != -1 else None,
Review Comment:
CC @mobuchowski @kacpermuda for review on this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2081096254
##########
airflow-core/src/airflow/executors/workloads.py:
##########
@@ -71,6 +71,8 @@ class TaskInstance(BaseModel):
context_carrier: dict | None = None
queued_dttm: datetime | None = None
+ log_url: str | None = None
Review Comment:
Basically we have all the information in task_runner.py to construct the uri, lets do it there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "ayush3singh (via GitHub)" <gi...@apache.org>.
ayush3singh commented on PR #50376:
URL: https://github.com/apache/airflow/pull/50376#issuecomment-2868608840
@amoghrajesh please review
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fixes issue RuntimeTaskInstance context does not contain log_url [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2083512761
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: ti.log_url if getattr(ti, "log_url", -1) != -1 else None,
Review Comment:
We can just do `getattr(ti, "log_url", None)`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "ayush3singh (via GitHub)" <gi...@apache.org>.
ayush3singh commented on code in PR #50376:
URL: https://github.com/apache/airflow/pull/50376#discussion_r2084054293
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -444,6 +444,7 @@ class TaskInstanceInfo(InfoJsonEncodable):
includes = ["duration", "try_number", "pool", "queued_dttm", "log_url"]
casts = {
+ "log_url": lambda ti: getattr(ti, "log_url", None),
Review Comment:
Hi @kacpermuda
without this , the below compatibility test fails:
provider distributions tests / Compat 3.0.0:P3.9
if we make this change then the Test succeeds as it will keep a place holder if not available in object ( in case of Airflow)
Keeping in `includes` list will add attribute only if original jaon hasvthe key, but in Airflow 3.0 its not present
Thanks for review comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implementing `log_url` on RuntimeTaskInstance in task sdk [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #50376:
URL: https://github.com/apache/airflow/pull/50376#issuecomment-2921831475
Oh ok, @ayush3singh feel free to create a new PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org