You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "frank-ye (via GitHub)" <gi...@apache.org> on 2026/10/13 18:37:13 UTC
[PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
frank-ye opened a new pull request, #74666:
URL: https://github.com/apache/airflow/pull/74666
## Description
`AirflowRunFacet.task` now carries two new keys, `inlets_list` and `outlets_list`, with the same assets as `inlets` / `outlets` but as real JSON lists instead of a Python `str(list)` repr. Runtime-emitted asset events (from `outlet_events`) are included, same as in `outlets`.
Values in `extra` that JSON can't encode are cast with `str()` (e.g. `Decimal("10")` -> `"10"`, a date -> `"2026-01-02"`), and a `str` Enum uri is sent as its value. If conversion still fails, a warning is logged and the key is omitted, so the rest of the facet is unaffected.
The existing `inlets` / `outlets` string keys are unchanged, so current consumers keep working.
Before / after for one outlet whose `extra` has a quoted etag and a bool:
"outlets": "[{'uri': 's3://b/o.csv', 'extra': {'etag': '\"0xABC\"', 'ok': True}, 'type': 'asset'}]"
"outlets_list": [{"uri": "s3://b/o.csv", "extra": {"etag": "\"0xABC\"", "ok": true}, "type": "asset"}]
## Motivation
`inlets` / `outlets` are sent as a Python repr string, so consumers have to parse Python literal syntax to get the asset uris. Since #65727 added runtime asset events, `extra` routinely holds free-form user payloads, and values like a string containing `"` or `'`, `True`, `None` or `Decimal(...)` make the usual "swap quotes and parse as JSON" approach fail for the whole list. A consumer then silently drops every outlet of the task. Sending a real list removes the need to parse a repr at all.
related: #65727
## Testing
- [X] Unit tests added / updated
- `extra` with quotes, apostrophe, `True`, `None` -> preserved as-is in `outlets_list`
- runtime asset events with `Decimal` / `date` in `extra` -> stringified, alias events keep `source_alias`
- task with no inlets / outlets -> empty lists
- non-encodable value (tuple dict key) -> warning logged, key omitted
- [X] Serialized an `AirflowRunFacet` through the OpenLineage client's `Serde.to_json` and confirmed `outlets_list` comes out as a JSON array with all values intact
## Notes for Reviewer
- The lists are written onto the dict after `InfoJsonEncodable.__init__`, because `_cast_basic_types` turns any list into a repr string. Happy to move this into a generic "raw fields" option on the base class if preferred.
- Key naming (`*_list`) is open to suggestions.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #74666:
URL: https://github.com/apache/airflow/pull/74666#discussion_r4271692120
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
Review Comment:
I like the above code, we can probably move it into the _extend_fields as well, just not add them to self._fields so that the _cast_basic_types is not applied to them (with a clear comment on why we're doing it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "frank-ye (via GitHub)" <gi...@apache.org>.
frank-ye commented on PR #74666:
URL: https://github.com/apache/airflow/pull/74666#issuecomment-6179322899
> Can we also please make sure to adjust openlineage system tests files by adding these new keys where they belong? Look at `tests/system/openlineage/expected_events` and all files that have inlets / outlets specified there.
Good call, added `inlets_list` / `outlets_list` to all the expected_events files in e31f1751. AF2 ones skip `extra` since the client drops None values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski merged PR #74666:
URL: https://github.com/apache/airflow/pull/74666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #74666:
URL: https://github.com/apache/airflow/pull/74666#issuecomment-6170137910
Can we also please make sure to adjust openlineage system tests files by adding these new keys where they belong? Look at `tests/system/openlineage/expected_events` and all files that have inlets / outlets specified there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "frank-ye (via GitHub)" <gi...@apache.org>.
frank-ye commented on code in PR #74666:
URL: https://github.com/apache/airflow/pull/74666#discussion_r4276678119
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -2203,6 +2209,15 @@ class AirflowContextDeprecationWarning(UserWarning):
return item
+def _convert_to_json_safe_list(value: Collection, name: str) -> list | None:
+ """Return ``value`` as a list of JSON-native types, casting non-serializable values with ``str()``."""
+ try:
+ return json.loads(json.dumps(list(value), default=str))
+ except (TypeError, ValueError) as e:
+ log.warning("Unable to serialize task `%s` to a JSON list: %s", name, e)
+ return None
+
+
Review Comment:
Kept it in init for now, setting the keys in `_extend_fields` only works because `dict.__init__` doesn't clear them, felt a bit too subtle tbh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "frank-ye (via GitHub)" <gi...@apache.org>.
frank-ye commented on code in PR #74666:
URL: https://github.com/apache/airflow/pull/74666#discussion_r4276687976
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -2203,6 +2209,15 @@ class AirflowContextDeprecationWarning(UserWarning):
return item
+def _convert_to_json_safe_list(value: Collection, name: str) -> list | None:
+ """Return ``value`` as a list of JSON-native types, casting non-serializable values with ``str()``."""
+ try:
+ return json.loads(json.dumps(list(value), default=str))
+ except (TypeError, ValueError) as e:
+ log.warning("Unable to serialize task `%s` to a JSON list: %s", name, e)
+ return None
+
+
Review Comment:
Makes sense, switched to `skipkeys=True` and dropped the `list()` in e31f1751 - a bad key now only loses that key instead of the whole list thanks for the catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #74666:
URL: https://github.com/apache/airflow/pull/74666#discussion_r4271729161
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -2203,6 +2209,15 @@ class AirflowContextDeprecationWarning(UserWarning):
return item
+def _convert_to_json_safe_list(value: Collection, name: str) -> list | None:
+ """Return ``value`` as a list of JSON-native types, casting non-serializable values with ``str()``."""
+ try:
+ return json.loads(json.dumps(list(value), default=str))
+ except (TypeError, ValueError) as e:
+ log.warning("Unable to serialize task `%s` to a JSON list: %s", name, e)
+ return None
+
+
Review Comment:
list() is probably not needed here, they are already lists, but I get that the helper is more generic. Also, we can think about using skipkeys=True instead of discarding the whole value here, but that's your choice - if we want to keep any outlets/inlets just without broken keys or show that its missing, I think the first option makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #74666:
URL: https://github.com/apache/airflow/pull/74666#discussion_r4271692120
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
Review Comment:
I like the above code, we can probably move it into the _extend_fields as well, just not add them to self._fields so that the _cast_basic_types is not applied to them (with a clear comment on why we're doing it). But keeping in init is also fine, just nut where other `custom` code is now for outlets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "frank-ye (via GitHub)" <gi...@apache.org>.
frank-ye commented on code in PR #74666:
URL: https://github.com/apache/airflow/pull/74666#discussion_r4276678119
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -2203,6 +2209,15 @@ class AirflowContextDeprecationWarning(UserWarning):
return item
+def _convert_to_json_safe_list(value: Collection, name: str) -> list | None:
+ """Return ``value`` as a list of JSON-native types, casting non-serializable values with ``str()``."""
+ try:
+ return json.loads(json.dumps(list(value), default=str))
+ except (TypeError, ValueError) as e:
+ log.warning("Unable to serialize task `%s` to a JSON list: %s", name, e)
+ return None
+
+
Review Comment:
Kept it in init for now, setting the keys in `_extend_fields` only works because `dict.__init__` doesn't clear them, felt a bit too subtle tbh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "frank-ye (via GitHub)" <gi...@apache.org>.
frank-ye commented on code in PR #74666:
URL: https://github.com/apache/airflow/pull/74666#discussion_r4276684348
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
Review Comment:
Kept it in init for now, setting the keys in `_extend_fields` only works because `dict.__init__` doesn't clear them, felt a bit too subtle
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
Review Comment:
Kept it in init for now, setting the keys in `_extend_fields` only works because `dict.__init__` doesn't clear them, felt a bit too subtle tbh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Send task inlets and outlets as JSON lists in OpenLineage facet [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #74666:
URL: https://github.com/apache/airflow/pull/74666#issuecomment-6266056608
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org