You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "alexdrydew (via GitHub)" <gi...@apache.org> on 2024/04/04 11:07:24 UTC

[I] expand_kwargs [airflow]

alexdrydew opened a new issue, #38745:
URL: https://github.com/apache/airflow/issues/38745

   ### Apache Airflow version
   
   2.8.4
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   It looks like expand_kwargs does not preserve templates that are stored in xcom
   
   ### What you think should happen instead?
   
   I believe in this case templates should not be rendered as this is the behaviour of basic `PlainXComArg.resolve` and simple `.expand` as well
   
   ### How to reproduce
   
   ```python
   import datetime
   
   from airflow.decorators import dag, task
   
   
   @task
   def some_task():
       return [{"value": "{{ foo }}"}]
   
   
   @task
   def print_task(value):
       print(value)
   
   
   @dag(
       schedule=None,
       start_date=datetime.datetime(2024, 1, 19),
       catchup=False,
   )
   def pipeline():
       print_task.expand_kwargs(some_task())
   
   
   pipeline().test()
   ```
   
   This fails with:
   ```
   jinja2.exceptions.UndefinedError: 'foo' is undefined
   ```
   
   ### Operating System
   
   macOS 14.2
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2531662626

   @PowerToThePeople111  I am not sure what parameters you are trying to pass, but if you see all that problems the easiest thing is to make sure you have a code that serializes your parameters to string (`string_args` option of the PythonVenvOperator) and add a code to your venv operator to deserialize those parameters from those serialized args.
   
   I believe PythonVirtualenvOperator does not support template in string_args, but there is - I think no particular reasons it could no, so you can create your own `MyPythonVirtualenvOperator(PythonVirtualenVOperator)` where you could overide template strings similarly to what is done in the PVO:
   
   ```
       template_fields: Sequence[str] = tuple(
           {"requirements", "index_urls", "venv_cache_path"}.union(PythonOperator.template_fields)
       )
   ```
   
   And once you can confirm it works for your case - contributing template fields change back to Airlfow might be a good idea.
   
   You have to rememver that even if operators in Airflow do not support something, it's equally easy to extend them and implement what you need as it is to write your DAG - so pretty natural thing to do when you need somethig is to extend the operators and then possibly even contribute it back.
   
   If you still are unsure about what you can do , I suggest you open a new discussion with a bit more detailed information of what parameters you are passing and why string_args cannot be used - it might or might not be similar discussion to this one - but the more details and examples you provide in a "fresh" discussion, they higher chance is that you can grab attention of someone who will decide to spend their free time on helping you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2037741292

   Oh... you mean it should be remaining the same. That is interesting behaviour TBH


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2039774537

   > In my opinion, the main problem with having templates being rendered in this case is that now we need to sanitize user data when processing it in expanded tasks (I didn't think of that at the beginning, but it is probably also a security issue in this case). When using plain tasks you generally don't need to be concerned whether the data you are processing can be harmful outside of the user code inside decorated python function context. Personally I think that would be unfortunate if this becomes a default behaviour for expand as well
   
   I don't think there are any significant security implications here. 
   
   The only code that can really get executed and templates that are coming in the DAGs come from DAG Authors. There is no code that can be executed that can be supplied (as far as I know) by someone else (i.e. UI users - except those users that have "Connection Editing" capabilities. And this is all within Airflow Security Model: https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html  - DAG authors already have capabilties  to execute anything they want and there is no way nor intention to block them from doing so (this can be controlled and verified at code review / static checks on DAGs outside of Airflow).
   
   Unless there is a scenario that other airflow users can provide a code that can be executed in the context of workers, I do not see how template expansion could go outside of the security model we already have and know.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-3782868404

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "alexdrydew (via GitHub)" <gi...@apache.org>.
alexdrydew commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2039259618

   > If we don't do this, we won't be able to use templates in mapped tasks, which is a big limitation for this great feature, so if it's not the same behavior as expand, then we need to fix expand method; I'll take a look at it when I have some free time.
   
   In my opinion, the main problem with having templates being rendered in this case is that now we need to sanitize user data when processing it in expanded tasks (I didn't think of that at the beginning, but it is probably also a security issue in this case). When using plain tasks you generally don't need to be concerned whether the data you are processing can be harmful outside of the user code inside decorated python function context. Personally I think that would be unfortunate if this becomes a default behaviour for `expand` as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2041528618

   > in this case data author could include {{ var.value.get('SOME_SECRET') }} template and get access to the variable if the target storage is available for them
   
   Yes. But you could do that with any code. DAG author could call `self.render_template(`<EXTERNAL DATA>`, context) ` and store result somewhere  as well. I think even the name of the method is pretty explicit that it is about expanding kwargs of a task. The signature of the method is:
   
   ```
   def expand_kwargs(self, kwargs: OperatorExpandKwargsArgument, *, ...).
   ```
   
   I don't see a danger of accidentally passing untrusted data there (but maybe I am missing something?)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "PowerToThePeople111 (via GitHub)" <gi...@apache.org>.
PowerToThePeople111 commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2531243081

   Are there any updates on this?
   I want to pass a parameter to a mapped PythonVirtualenvOperator task and although I tried many things, I seem to be unable to do so.
   
   I tried:
   1) providing the dag-parameter in the `op_kwargs`-parameter of the `partial` function of `PythonVirtualenvOperator` constructor while expanding with the other variable `op_kwargs`-parameters. This did not work saying that i supplied illegal arguments. Probably because you can not supply `op_kwargs` in partial with some keys and expand with `op_kwargs` with other keys. 
   2) setting a Variable in the start node of the dag with the value i could get there from the params-dictionary. Then in the mapped task, I tried to access this variable using `Variable.get`, but this needs me to add airflow to the tasks PythonVirtualenvOperator's requirements. But if I add airflow to the requirements, i have problems using the tensorflow package due to dependency conflicts which I need to have present. If I try to read the variable using a jinja-template it will fill it at the very beginning of the dag, before i can set it with the current parameter.
   3) Something similar as 2) but trying to directly access the context by using `get_current_context`.
   
   I feel like i am running out of options.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "alexdrydew (via GitHub)" <gi...@apache.org>.
alexdrydew commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2041073791

   My concern was primarily about processing untrusted data from external sources in DAGs: it seems malicious data can be used to steal secrets in some cases:
   ```python
   @dag(...)
   def pipeline():
       data = download_parameters_from_s3()
       transformed_data = transform.expand_kwargs(data)
       upload_to_s3(transformed_data)
   ```
   in this case data author could include `{{ var.value.get('SOME_SECRET') }}` template and get access to the variable if the target storage is available for them. I understand that this case is probably out of scope of the airflow security model but the way how plain TaskFlow-style tasks communicate using XCom allows to process untrusted data in this way.
   
   But not to change focus: my main concern is that even if we don't return processed untrusted data to potentially malicious user back we still need to sanitize inputs specifically for `expand_kwargs` in order not to fail while processing data that may contain template-like syntax (e.g. parsed webpage)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "alexdrydew (via GitHub)" <gi...@apache.org>.
alexdrydew commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2089076012

   > I think even the name of the method is pretty explicit that it is about expanding kwargs of a task.
   
   But "expanding kwargs" does not describe whether templates will be rendered or not if some of the passed keyword arguments happen to be template strings. A valid point here could be that it is the default behaviour for how airflow treats arguments for the tasks but it is actually not the case (at least for usual TaskFlow tasks).
   
   Case 1, template is defined during DAG import:
   ```python
   @task  
   def my_task(dag_id: str):  
       assert dag_id == 'my_dag'  
     
   @dag(  
       schedule=None,  
       start_date=datetime.datetime(2024, 1, 19),  
       catchup=False,  
   )  
   def my_dag():  
       my_task(dag_id='{{ dag.dag_id }}')
   ```
   Case 2, template string is retrieved from XComArg:
   ```python
   @task  
   def construct_template() -> str:  
       return '{{ dag.dag_id }}'  
     
   @task  
   def my_task(dag_id: str):  
       assert dag_id == '{{ dag.dag_id }}'  
     
   @dag(  
       schedule=None,  
       start_date=datetime.datetime(2024, 1, 19),  
       catchup=False,  
   )  
   def my_dag():  
       my_task(dag_id=construct_template())
   ```
   
   In our experience the fact that `expand_kwargs` resolves templates besides just unpacking value from XComArg to be passed as kwargs and mapping it to dynamic number of task instances was confusing and counterintuitive.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2038444630

   For more context, I updated the behavior of `expand_kwargs` in 2.7.1 by https://github.com/apache/airflow/pull/32272 to fix https://github.com/apache/airflow/issues/32260.
   
   If we don't do this, we won't be able to use templates in mapped tasks, which is a big limitation for this great feature, so if it's not the same behavior as `expand`, then we need to fix `expand` method; I'll take a look at it when I have some free time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "PowerToThePeople111 (via GitHub)" <gi...@apache.org>.
PowerToThePeople111 commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2531702516

   > @PowerToThePeople111 I am not sure what parameters you are trying to pass, but if you see all that problems the easiest thing is to make sure you have a code that serializes your parameters to string (`string_args` option of the PythonVenvOperator) and add a code to your venv operator to deserialize those parameters from those serialized args.
   > 
   > I believe PythonVirtualenvOperator does not support template in string_args, but there is - I think no particular reasons it could no, so you can create your own `MyPythonVirtualenvOperator(PythonVirtualenVOperator)` where you could overide template strings similarly to what is done in the PVO:
   > 
   > ```
   >     template_fields: Sequence[str] = tuple(
   >         {"requirements", "index_urls", "venv_cache_path"}.union(PythonOperator.template_fields)
   >     )
   > ```
   > 
   > And once you can confirm it works for your case - contributing template fields change back to Airlfow might be a good idea.
   > 
   > You have to rememver that even if operators in Airflow do not support something, it's equally easy to extend them and implement what you need as it is to write your DAG - so pretty natural thing to do when you need somethig is to extend the operators and then possibly even contribute it back.
   > 
   > If you still are unsure about what you can do , I suggest you open a new discussion with a bit more detailed information of what parameters you are passing and why string_args cannot be used - it might or might not be similar discussion to this one - but the more details and examples you provide in a "fresh" discussion, they higher chance is that you can grab attention of someone who will decide to spend their free time on helping you.
   
   @potiuk 
   Thank you for your reply! Yeah I guess I will try to read into the code to understand how to adapt everything. If I find a solution, I will post it here. In the worst case, I will just use an airflow variable that I have to set instead of a parameter (although I love those params a whole lot more for configuring certain dag runs).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "gnudiff (via GitHub)" <gi...@apache.org>.
gnudiff commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2247694070

   Is this the same issue that when creating task via dynamic task mapping, references to {{ var }} and {{ params }} get lost when templating, or should I opend a new case?
   
   Eg. I have a custom operator created as per Airflow guide:
   
   ```
   from airflow.models import BaseOperator
   
   class CreateXMLFromTemplate(BaseOperator):
       template_fields = ('xml_file',)
       template_ext = ('.xml','.html')
   
       def __init__(
       self,
       *,
       xml_file,
       output_file,
       parameters = None,
       **kwargs,
       ):
           super().__init__(**kwargs)
           self.xml_file=xml_file
           self.parameters = parameters
           self.output_file=output_file
       
       def execute(self,context):
           with open(self.output_file,'w') as F:
               F.write(self.xml_file)
           return self.output_file
   ```
   
   
   When called via usual tasks calls, the xml_file renders all {{ var.json .... }} and {{ params.field }} as expected:
   
   ```
       render_xml=CreateXMLFromTemplate(
       task_id='test_xml',
       xml_file='test.xml',
       output_file='/tmp/output.xml',
       parameters={ 'cargo':{ 'prep_date':'123123' } }
       )
   ```
   
   When called via dynamic task mapping, it loses access to both {{ var }} and {{ params }}:
   
   ```
   @task
   def output_dynamic_task_data(): 
       results=[]
       for x in data: 
          single_result={'prep_date': xxxx }
          params=.... construct params dict
          single_result['parameters']=params
          results.append(single_result)
       return results
       
   multi_output=output_dynamic_task_data()
   
   create_xml_task=CreateXMLFromTemplate.partial(
       task_id='data_to_xml', xml_file='test.xml').expand_kwargs(multi_output)
   ```
   
   Now the same CreateXMLFromTemplate operator when parsing the same xml file will come up with empty for {{ var }} and {{ params }}


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2036865801

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-2037638574

   > return [{"value": "{{ foo }}"}]
   
   return [{"value": "{{ 'foo' }}"}]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38745:
URL: https://github.com/apache/airflow/issues/38745#issuecomment-3685442888

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] expand_kwargs does not preserve templates [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #38745: expand_kwargs does not preserve templates
URL: https://github.com/apache/airflow/issues/38745


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org