You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ephraimbuddy (via GitHub)" <gi...@apache.org> on 2024/03/12 23:04:19 UTC
[PR] Prevent large objects from being stored in the RTIF [airflow]
ephraimbuddy opened a new pull request, #38094:
URL: https://github.com/apache/airflow/pull/38094
There's no control over the size of objects stored in the rendered taskinstance field. This PR adds control and enable users to be able to customize the size of data that can be stored in this field
closes: #28199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1535747980
##########
airflow/config_templates/config.yml:
##########
@@ -492,6 +492,14 @@ core:
type: string
example: ~
default: "Disabled"
+ max_templated_field_size:
+ description: |
+ The maximum size of the rendered template field in bytes. If the value to be stored in the
Review Comment:
Nit: not bytes, characters.
```suggestion
The maximum length of the rendered template field. If the value to be stored in the
```
##########
airflow/serialization/helpers.py:
##########
@@ -38,7 +39,19 @@ def is_jsonable(x):
else:
return True
+ max_size = conf.getint("core", "max_templated_field_size")
if not is_jsonable(template_field):
- return str(template_field)
+ serialized = str(template_field)
+ if len(serialized) > max_size:
+ return (
+ "Value removed due to size. "
+ "You can change this behaviour in [core]max_templated_field_size"
+ )
+ return serialized
else:
+ if template_field and len(str(template_field)) > max_size:
+ return (
+ "Value removed due to size. "
+ "You can change this behaviour in [core]max_templated_field_size"
+ )
Review Comment:
Since you are always comparing the str length, you don't have to do it in each branch, you can do it before the if.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #38094:
URL: https://github.com/apache/airflow/pull/38094#issuecomment-2015670623
> Nice. The prefix is cool.
>
> There is one worry with that U just wanted to mention it.
>
> I understand that we are masking secreates before serialization ? Just wanted to amke sure sure of it - it looks like from the code , but I wanted to make sure we are not showing half-unmasked secrets accidentally when we truncate the secrets.
The secrets are masked after serialization see: https://github.com/apache/airflow/blob/6a225ccb2a9b51c52df88368323993bcd017be0d/airflow/models/renderedtifields.py#L127. I have added a test and will also do manual testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1522605282
##########
airflow/config_templates/config.yml:
##########
@@ -481,6 +481,14 @@ core:
type: string
example: ~
default: "Disabled"
+ max_templated_field_size:
+ description: |
+ The maximum size of the rendered template field. If the value to be stored in the rendered template
Review Comment:
Can we add the size quatifier i.e number of bytes or characters or etc.?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #38094:
URL: https://github.com/apache/airflow/pull/38094#issuecomment-2015691919
> > Nice. The prefix is cool.
> > There is one worry with that U just wanted to mention it.
> > I understand that we are masking secreates before serialization ? Just wanted to amke sure sure of it - it looks like from the code , but I wanted to make sure we are not showing half-unmasked secrets accidentally when we truncate the secrets.
>
> The secrets are masked after serialization see:
>
> https://github.com/apache/airflow/blob/6a225ccb2a9b51c52df88368323993bcd017be0d/airflow/models/renderedtifields.py#L127
>
> . I have added a test and will also do manual testing
It doesn't work so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1535473486
##########
airflow/config_templates/config.yml:
##########
@@ -492,6 +492,14 @@ core:
type: string
example: ~
default: "Disabled"
+ max_templated_field_size:
+ description: |
+ The maximum size of the rendered template field in bytes. If the value to be stored in the
+ rendered template field exceeds this size, it's redacted.
+ version_added: 2.9.0
+ type: integer
+ example: ~
+ default: "1024"
Review Comment:
NIT. 1024 looks like rather low default value. I'd put it to 4096 (another very opinionated magic value here) but also I'd mention that as significant newsfragment as it might make big difference for some people..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "travis-cook-sfdc (via GitHub)" <gi...@apache.org>.
travis-cook-sfdc commented on PR #38094:
URL: https://github.com/apache/airflow/pull/38094#issuecomment-2230086650
When a field gets truncated, is that _only_ for the purpose of rendering it the UI? Or will it also get truncated when it gets executed?
The release notes on this topic are not clear and I'm seeing the truncation occur in both `serialized_dag` and `rendered_task_fields`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1537106372
##########
tests/models/test_renderedtifields.py:
##########
@@ -148,6 +170,49 @@ def test_get_templated_fields(self, templated_field, expected_rendered_field, da
# Fetching them will return None
assert RTIF.get_templated_fields(ti=ti2) is None
+ def test_secrets_are_masked_when_large_string(self, dag_maker):
+ """
+ Test that secrets are masked when the templated field is a large string
+ """
+ Variable.set(
+ key="api_key",
+ value="Some very long secret with private information that asserts private is not in the rendered field"
Review Comment:
```suggestion
value="Api key secret should be masked as before"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1522245413
##########
airflow/serialization/helpers.py:
##########
@@ -37,7 +39,26 @@ def is_jsonable(x):
else:
return True
+ max_size = conf.getint("core", "max_templated_field_size")
+
if not is_jsonable(template_field):
+ if isinstance(template_field, (list, tuple)):
Review Comment:
Why does this only check the first item?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1525250988
##########
airflow/serialization/helpers.py:
##########
@@ -37,7 +39,26 @@ def is_jsonable(x):
else:
return True
+ max_size = conf.getint("core", "max_templated_field_size")
+
if not is_jsonable(template_field):
+ if isinstance(template_field, (list, tuple)):
Review Comment:
Just did
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #38094:
URL: https://github.com/apache/airflow/pull/38094#issuecomment-2012925956
I will like to include this in the next beta/rc, please review @uranusjr @jedcunningham @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1536707455
##########
airflow/serialization/helpers.py:
##########
@@ -38,7 +39,19 @@ def is_jsonable(x):
else:
return True
+ max_size = conf.getint("core", "max_templated_field_size")
if not is_jsonable(template_field):
- return str(template_field)
+ serialized = str(template_field)
+ if len(serialized) > max_size:
+ return (
+ "Value removed due to size. "
+ "You can change this behaviour in [core]max_templated_field_size"
+ )
+ return serialized
else:
+ if template_field and len(str(template_field)) > max_size:
+ return (
+ "Value removed due to size. "
+ "You can change this behaviour in [core]max_templated_field_size"
+ )
Review Comment:
Changing to doing it in separate branches cause we can't use `if` statement on dataframe. It gives a valueerror
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #38094:
URL: https://github.com/apache/airflow/pull/38094#issuecomment-2011670019
There's something wrong with MySQL around templated fields. cc @uranusjr .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1537110231
##########
airflow/serialization/helpers.py:
##########
@@ -38,7 +40,21 @@ def is_jsonable(x):
else:
return True
+ max_length = conf.getint("core", "max_templated_field_length")
+
if not is_jsonable(template_field):
+ if len(str(template_field)) > max_length:
+ rendered = redact(str(template_field), name)
+ return (
+ "Truncated. You can change this behaviour in [core]max_templated_field_length. "
+ f"{rendered[:max_length - 79]!r}... "
+ )
return str(template_field)
else:
+ if template_field and len(str(template_field)) > max_length:
+ rendered = redact(str(template_field), name)
Review Comment:
Do we want to use `json.dumps` here?
Also we should reduce the stringify calls (either `str` or `json.dumps`) to exactly once for each time this function is called. The call can be expensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy merged PR #38094:
URL: https://github.com/apache/airflow/pull/38094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1522615945
##########
airflow/serialization/helpers.py:
##########
@@ -37,7 +39,26 @@ def is_jsonable(x):
else:
return True
+ max_size = conf.getint("core", "max_templated_field_size")
+
if not is_jsonable(template_field):
+ if isinstance(template_field, (list, tuple)):
Review Comment:
I would suggest to calculate the overall object size (summing the all the elements sizes) and compare it with the max size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1535756076
##########
airflow/serialization/helpers.py:
##########
@@ -38,7 +39,19 @@ def is_jsonable(x):
else:
return True
+ max_size = conf.getint("core", "max_templated_field_size")
if not is_jsonable(template_field):
- return str(template_field)
+ serialized = str(template_field)
+ if len(serialized) > max_size:
+ return (
+ "Value removed due to size. "
Review Comment:
Should we truncate vs remove it completely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1535918174
##########
airflow/serialization/helpers.py:
##########
@@ -38,6 +39,13 @@ def is_jsonable(x):
else:
return True
+ max_length = conf.getint("core", "max_templated_field_length")
+
+ if template_field and len(str(template_field)) > max_length:
+ return (
+ f"{str(template_field)[:max_length-79]}... truncated. "
+ "You can change this behaviour in [core]max_templated_field_length"
Review Comment:
Let's do this as a prefix instead. It's more obvious if its at the beginning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1535778314
##########
airflow/serialization/helpers.py:
##########
@@ -38,7 +39,19 @@ def is_jsonable(x):
else:
return True
+ max_size = conf.getint("core", "max_templated_field_size")
if not is_jsonable(template_field):
- return str(template_field)
+ serialized = str(template_field)
+ if len(serialized) > max_size:
+ return (
+ "Value removed due to size. "
Review Comment:
My concern was the information about how to configure the size. Would it make sense to have a truncated data alongside the info about how to increase the config value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1535790800
##########
airflow/serialization/helpers.py:
##########
@@ -38,7 +39,19 @@ def is_jsonable(x):
else:
return True
+ max_size = conf.getint("core", "max_templated_field_size")
if not is_jsonable(template_field):
- return str(template_field)
+ serialized = str(template_field)
+ if len(serialized) > max_size:
+ return (
+ "Value removed due to size. "
Review Comment:
Seems reasonable to me, as a prefix so it's visible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1537105429
##########
tests/models/test_renderedtifields.py:
##########
@@ -62,6 +64,18 @@ def __ne__(self, other):
return not self.__eq__(other)
+class LargeStrObject:
+ def __init__(self):
+ self.a = "a" * 5000
+
+ def __str__(self):
+ return self.a
+
+
+max_length = conf.getint("core", "max_templated_field_length")
+LargeStr = "a" * 5000
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent large objects from being stored in the RTIF [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #38094:
URL: https://github.com/apache/airflow/pull/38094#discussion_r1537127130
##########
airflow/serialization/helpers.py:
##########
@@ -38,7 +40,21 @@ def is_jsonable(x):
else:
return True
+ max_length = conf.getint("core", "max_templated_field_length")
+
if not is_jsonable(template_field):
+ if len(str(template_field)) > max_length:
+ rendered = redact(str(template_field), name)
+ return (
+ "Truncated. You can change this behaviour in [core]max_templated_field_length. "
+ f"{rendered[:max_length - 79]!r}... "
+ )
return str(template_field)
else:
+ if template_field and len(str(template_field)) > max_length:
+ rendered = redact(str(template_field), name)
Review Comment:
Using json.dumps brought up some extra quotes. Reduced str usage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org