You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mobuchowski (via GitHub)" <gi...@apache.org> on 2024/07/03 15:50:44 UTC
[PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
mobuchowski opened a new pull request, #40589:
URL: https://github.com/apache/airflow/pull/40589
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1678000068
##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -153,3 +153,10 @@ config:
example: ~
type: integer
version_added: 1.9.0
+ include_full_task_info:
+ description: |
+ If true, OpenLineage event will include full task info - potentially containing large fields.
Review Comment:
@kaxil It's task dependent. I've attached two events, one with full task info enabled, one without, for S3ToGCSOperator:
[af_with_full_info.json](https://github.com/user-attachments/files/16236609/af_with_full_info.json)
[af_without_full_task_info.json](https://github.com/user-attachments/files/16236610/af_without_full_task_info.json)
I've also added section to the user doc, where we explain particular conf variables: https://github.com/apache/airflow/pull/40589/files#diff-c1cb5b68b135a7ce9b52d925007cbbdcdaf03ec5e72871abcb1354561738b239
link to prod one: https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1668771486
##########
tests/providers/openlineage/test_conf.py:
##########
@@ -487,3 +489,20 @@ def test_dag_state_change_process_pool_size(var_string, expected):
with conf_vars({(_CONFIG_SECTION, _CONFIG_OPTION_DAG_STATE_CHANGE_PROCESS_POOL_SIZE): var_string}):
result = dag_state_change_process_pool_size()
assert result == expected
+
+
+@pytest.mark.parametrize(
+ ("var", "expected"),
+ (
+ ("False", False),
+ ("True", True),
+ ("t", True),
+ ("a", False),
+ ("asdf", False),
+ ("31", False),
+ ("true", True),
Review Comment:
```suggestion
("true", True),
("", False),
(" ", False),
(None, False)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1668920281
##########
airflow/providers/openlineage/conf.py:
##########
@@ -145,3 +145,8 @@ def execution_timeout() -> int:
"""[openlineage] execution_timeout."""
option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
return _safe_int_convert(str(option).strip(), default=10)
+
+
+@cache
+def include_full_task_info() -> bool:
+ return _is_true(conf.get(_CONFIG_SECTION, "include_full_task_info", fallback="False"))
Review Comment:
we have `getboolean` ... I think it can/should be used here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski merged PR #40589:
URL: https://github.com/apache/airflow/pull/40589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1668986995
##########
airflow/providers/openlineage/conf.py:
##########
@@ -145,3 +145,8 @@ def execution_timeout() -> int:
"""[openlineage] execution_timeout."""
option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
return _safe_int_convert(str(option).strip(), default=10)
+
+
+@cache
+def include_full_task_info() -> bool:
+ return _is_true(conf.get(_CONFIG_SECTION, "include_full_task_info", fallback="False"))
Review Comment:
Yes, probably it could, but i think we wanted to prevent breaking Airflow / raising errors by all means, so we put this explanation in the module docstring:
```
To prevent errors caused by invalid user-provided configuration values, we use ``conf.get()``
to fetch values as strings and perform safe conversions using custom functions.
Any invalid configuration values should be treated as incorrect and replaced with default values.
For example, if the default for boolean ``custom_ol_var`` is False, any non-true value provided:
``"asdf"``, ``12345``, ``{"key": 1}`` or empty string, will result in False being used.
By using default values for invalid configuration values, we ensure that the configurations are handled
safely, preventing potential runtime errors due to conversion issues.
```
Does it make sense or do you see any other viable way to do it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1677192787
##########
airflow/providers/openlineage/utils/utils.py:
##########
@@ -269,6 +269,25 @@ class TaskInfo(InfoJsonEncodable):
}
+class TaskInfoComplete(TaskInfo):
+ """Defines encoding BaseOperator/AbstractOperator object to JSON used when user enables full task info."""
+
+ includes = []
+ excludes = [
+ "_BaseOperator__instantiated",
+ "_dag",
+ "_hook",
+ "_log",
+ "_outlets",
+ "_inlets",
+ "_lock_for_execution",
+ "handler",
+ "params",
+ "python_callable",
+ "retry_delay",
Review Comment:
Yeah, but do we gain something by adding those? They were added after checking if it makes sense - the point for this PR is to enable serialization for those that we _don't know_ if are okay. I think there's no point for that `python_callable` there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1677718815
##########
airflow/providers/openlineage/conf.py:
##########
@@ -145,3 +145,8 @@ def execution_timeout() -> int:
"""[openlineage] execution_timeout."""
option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
return _safe_int_convert(str(option).strip(), default=10)
+
+
+@cache
+def include_full_task_info() -> bool:
+ return _is_true(conf.get(_CONFIG_SECTION, "include_full_task_info", fallback="False"))
Review Comment:
Upd: @kacpermuda prepared better PR at https://github.com/apache/airflow/pull/40790
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1677203689
##########
tests/providers/openlineage/test_conf.py:
##########
@@ -487,3 +489,20 @@ def test_dag_state_change_process_pool_size(var_string, expected):
with conf_vars({(_CONFIG_SECTION, _CONFIG_OPTION_DAG_STATE_CHANGE_PROCESS_POOL_SIZE): var_string}):
result = dag_state_change_process_pool_size()
assert result == expected
+
+
+@pytest.mark.parametrize(
+ ("var", "expected"),
+ (
+ ("False", False),
+ ("True", True),
+ ("t", True),
+ ("a", False),
+ ("asdf", False),
+ ("31", False),
+ ("true", True),
Review Comment:
Split the tests due to changes discussed below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1668897741
##########
airflow/providers/openlineage/utils/utils.py:
##########
@@ -269,6 +269,25 @@ class TaskInfo(InfoJsonEncodable):
}
+class TaskInfoComplete(TaskInfo):
+ """Defines encoding BaseOperator/AbstractOperator object to JSON used when user enables full task info."""
+
+ includes = []
+ excludes = [
+ "_BaseOperator__instantiated",
+ "_dag",
+ "_hook",
+ "_log",
+ "_outlets",
+ "_inlets",
+ "_lock_for_execution",
+ "handler",
+ "params",
+ "python_callable",
+ "retry_delay",
Review Comment:
I think we can remove some (suggestion below) or even all stuff from here. It'll not be serialized anyway, so we'll get e.g. `python_callable:"<<non-serializable: function>>"`. No strong opinion on this one, but I don't see why we should exclude anything if it's turned on explicitly now.
```suggestion
"_BaseOperator__instantiated",
"_dag",
"_hook",
"_log",
"_outlets",
"_inlets",
"_lock_for_execution",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1677203646
##########
airflow/providers/openlineage/conf.py:
##########
@@ -145,3 +145,8 @@ def execution_timeout() -> int:
"""[openlineage] execution_timeout."""
option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
return _safe_int_convert(str(option).strip(), default=10)
+
+
+@cache
+def include_full_task_info() -> bool:
+ return _is_true(conf.get(_CONFIG_SECTION, "include_full_task_info", fallback="False"))
Review Comment:
Ok, I've updated the PR to use `getboolean` consistently with rest of Airflow. I've converted rest in followup PR: https://github.com/apache/airflow/pull/40780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1669047207
##########
airflow/providers/openlineage/conf.py:
##########
@@ -145,3 +145,8 @@ def execution_timeout() -> int:
"""[openlineage] execution_timeout."""
option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
return _safe_int_convert(str(option).strip(), default=10)
+
+
+@cache
+def include_full_task_info() -> bool:
+ return _is_true(conf.get(_CONFIG_SECTION, "include_full_task_info", fallback="False"))
Review Comment:
Yes, what you're saying makes sense and it probably would be even easier for the user to debug (the whole Airflow stops working unless you provide a valid value). I don't know, i thought our approach makes sense but maybe we should change it back. Any thoughts @mobuchowski @JDarDagran ? I think I proposed it as we tried as hard as possible not to break anything in Airflow deployments, even if somebody provided an OL config with a typo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1677865645
##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -153,3 +153,10 @@ config:
example: ~
type: integer
version_added: 1.9.0
+ include_full_task_info:
+ description: |
+ If true, OpenLineage event will include full task info - potentially containing large fields.
Review Comment:
Could you add an example of what "full task info" means? What all is included?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1668920281
##########
airflow/providers/openlineage/conf.py:
##########
@@ -145,3 +145,8 @@ def execution_timeout() -> int:
"""[openlineage] execution_timeout."""
option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
return _safe_int_convert(str(option).strip(), default=10)
+
+
+@cache
+def include_full_task_info() -> bool:
+ return _is_true(conf.get(_CONFIG_SECTION, "include_full_task_info", fallback="False"))
Review Comment:
we have `getboolean` ... I think it can be used here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1669006910
##########
airflow/providers/openlineage/conf.py:
##########
@@ -145,3 +145,8 @@ def execution_timeout() -> int:
"""[openlineage] execution_timeout."""
option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
return _safe_int_convert(str(option).strip(), default=10)
+
+
+@cache
+def include_full_task_info() -> bool:
+ return _is_true(conf.get(_CONFIG_SECTION, "include_full_task_info", fallback="False"))
Review Comment:
BTW. I think the behaviour where invalid values are silently converted to default is a bad idea. Imagine this typo:
```
include_full_task_info: trve
```
Which might be difficult to spot depending on the font you are using in your IDE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1669002851
##########
airflow/providers/openlineage/conf.py:
##########
@@ -145,3 +145,8 @@ def execution_timeout() -> int:
"""[openlineage] execution_timeout."""
option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
return _safe_int_convert(str(option).strip(), default=10)
+
+
+@cache
+def include_full_task_info() -> bool:
+ return _is_true(conf.get(_CONFIG_SECTION, "include_full_task_info", fallback="False"))
Review Comment:
Well. It makes a bit inconsistent behaviour with other boolean flags, which might be surprising for the users. And if there is no good reason to diverge from the common approach, I think consistency trumps convenience (if you think the approach you proposed is a bit more convenient in some cases):
Is there a good reason why the defaulkt `getboolean` behaviour is a problem and can't be used ?
I mean this:
```python
def getboolean(self, section: str, key: str, **kwargs) -> bool: # type: ignore[override]
val = str(self.get(section, key, _extra_stacklevel=1, **kwargs)).lower().strip()
if "#" in val:
val = val.split("#")[0].strip()
if val in ("t", "true", "1"):
return True
elif val in ("f", "false", "0"):
return False
else:
raise AirflowConfigException(
f'Failed to convert value to bool. Please check "{key}" key in "{section}" section. '
f'Current value: "{val}".'
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1669002851
##########
airflow/providers/openlineage/conf.py:
##########
@@ -145,3 +145,8 @@ def execution_timeout() -> int:
"""[openlineage] execution_timeout."""
option = conf.get(_CONFIG_SECTION, "execution_timeout", fallback="")
return _safe_int_convert(str(option).strip(), default=10)
+
+
+@cache
+def include_full_task_info() -> bool:
+ return _is_true(conf.get(_CONFIG_SECTION, "include_full_task_info", fallback="False"))
Review Comment:
Well. It makes a bit inconsistent behaviour with other boolean flags, which might be surprising for the users. And if there is no good reason to diverge from the common approach, I think consistency trumps convenience (if you think the approach you proposed is a bit more convenient in some cases):
Is there a good reason why the common `getboolean` behaviour is a problem and can't be used ?
I mean this:
```python
def getboolean(self, section: str, key: str, **kwargs) -> bool: # type: ignore[override]
val = str(self.get(section, key, _extra_stacklevel=1, **kwargs)).lower().strip()
if "#" in val:
val = val.split("#")[0].strip()
if val in ("t", "true", "1"):
return True
elif val in ("f", "false", "0"):
return False
else:
raise AirflowConfigException(
f'Failed to convert value to bool. Please check "{key}" key in "{section}" section. '
f'Current value: "{val}".'
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1677871482
##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -153,3 +153,10 @@ config:
example: ~
type: integer
version_added: 1.9.0
+ include_full_task_info:
+ description: |
+ If true, OpenLineage event will include full task info - potentially containing large fields.
Review Comment:
Are there any high-level docs on what info can be gathered in the docs? and how users can control it apart from the config doc?
I was assuming something over here: https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/structure.html but didn't find it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] openlineage: add config to include 'full' task info based on conf setting [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #40589:
URL: https://github.com/apache/airflow/pull/40589#discussion_r1678023229
##########
docs/apache-airflow-providers-openlineage/guides/user.rst:
##########
@@ -246,6 +246,28 @@ full import paths of Airflow Operators to disable as ``disabled_for_operators``
AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS='airflow.operators.bash.BashOperator;airflow.operators.python.PythonOperator'
+Full Task Info
+^^^^^^^^^^^^^^
+
+By default, OpenLineage integration's AirflowRunFacet - attached on START event for every task instance event - does
+not contain full serialized task information (parameters to given operator), but only includes selected parameters.
+
+However, we allow users to set OpenLineage integration to include full task information. By doing this, rather than
+serializing only few known attributes, we operate in exclude mode - and exclude certain non-serializable elements.
+
+.. code-block:: ini
+
+ [openlineage]
+ transport = {"type": "http", "url": "http://example.com:5000", "endpoint": "api/v1/lineage"}
+ include_full_task_info = true
+
+``AIRFLOW__OPENLINEAGE__INCLUDE_FULL_TASK_INFO`` environment variable is an equivalent.
+
+.. warning::
+
+ By setting this variable to true, OpenLineage integration does not control the size of event you sent. It can potentially include elements weighting megabytes in size, depending on the size of data you pass to the task.
Review Comment:
```suggestion
By setting this variable to true, OpenLineage integration does not control the size of event you sent. It can potentially include elements that are megabytes in size or larger, depending on the size of data you pass to the task.
```
##########
docs/apache-airflow-providers-openlineage/guides/user.rst:
##########
@@ -246,6 +246,28 @@ full import paths of Airflow Operators to disable as ``disabled_for_operators``
AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS='airflow.operators.bash.BashOperator;airflow.operators.python.PythonOperator'
+Full Task Info
+^^^^^^^^^^^^^^
+
+By default, OpenLineage integration's AirflowRunFacet - attached on START event for every task instance event - does
+not contain full serialized task information (parameters to given operator), but only includes selected parameters.
Review Comment:
```suggestion
not contain full serialized task information (parameters to given operator), but only includes select parameters.
```
##########
docs/apache-airflow-providers-openlineage/guides/user.rst:
##########
@@ -246,6 +246,28 @@ full import paths of Airflow Operators to disable as ``disabled_for_operators``
AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS='airflow.operators.bash.BashOperator;airflow.operators.python.PythonOperator'
+Full Task Info
+^^^^^^^^^^^^^^
+
+By default, OpenLineage integration's AirflowRunFacet - attached on START event for every task instance event - does
+not contain full serialized task information (parameters to given operator), but only includes selected parameters.
+
+However, we allow users to set OpenLineage integration to include full task information. By doing this, rather than
+serializing only few known attributes, we operate in exclude mode - and exclude certain non-serializable elements.
Review Comment:
```suggestion
serializing only a few known attributes, we exclude certain non-serializable elements and send everything else.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org