You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/25 07:47:22 UTC
[GitHub] [airflow] alexInhert opened a new issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
alexInhert opened a new issue #16037:
URL: https://github.com/apache/airflow/issues/16037
Currently the operator allows to set requirement as list that needs to be hard coded.
It would be nice if airflow can support reading from file directly (something similar to how operators read sql file)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847735311
How about this (I know it's not perfect, but I think it should do the job):
```
requiremetns = (lambda: open("requirements.txt").readlines())()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847797452
I thinkl it the same feature we are talking about :).
The "template_ext" works in the way that it will read the file as string when the parameter has the extension registered there and it will automatically process the file trough the template engine, so you can add jinja-variables from context /xcom variables in the file. It's nuilt-in feature in BaseOperator
All that you need is in this case to have the right extension (".txt") in the value of the parameter.
```
requirements="[<path>/]reqiurements.txt"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847744371
Agree. Having a good, templated parameter could work. Now when I think obout it might actually be as simple as adding this to the operator and handling the case where `requirements` is also a string and properly using it with `-r` by the operator in this case:
```
template_fields = ("requirement",)
template_ext = (".txt",)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847999179
> I think in your use case the whole requirement file is in fact not a requirement.txt but a value pushed to xcom by previous task.
No. I am talking about "requirement.txt" file in which some values might be provided by XCom.
I can imagine this kind of `requirement.txt` file for example:
```
dependency1==1.0.0
dependency2==2.0.0
dependency3=={{ code to retrieve depedency version from xcom}}
dependency4==1.1.0
{{- % if retrireve deployment == 'production' % }}
# only needed in production deployment
dependency5==5.0.0
{{- % endif % }}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847799258
If the parameter is not string or does not have the extension it is processed as usual (i.e. passing list of strings will continue to work - but it will additionally process the strings through template engine (since "requirements" will be in "template_fields" tuple) so you could also add jinja variables in the strings in the array and they will be interpolated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847739889
The requirements file format supports some things that can’t be passed into the command line, so the open-readline approach can only partially solve the issue. Making things worse, the file format is internal to pip, so there’s no way for the user to parse it on their own, the file must be passed into pip. So some way to support this would be nice. (I think I read another issue on this before…?)
I believe `requirements=["-r", "requireements.txt]` will work. Pretty hacky though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vijaykiran commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
vijaykiran commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-877337120
I'd like to pick this (as part of the contributors-workshop)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexInhert commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
alexInhert commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847814829
I think in your use case the whole requirement file is in fact not a requirement.txt but a value pushed to xcom by previous task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #16037:
URL: https://github.com/apache/airflow/issues/16037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rounakdatta commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
rounakdatta commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-890093215
Hello @potiuk @alexInhert, I have tried an implementation of the above discussions, can you review once? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #16037:
URL: https://github.com/apache/airflow/issues/16037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847744371
Agree. Having a good, templated parameter could work. Now when I think obout it might actually be as simple as adding this to the operator and handling the case where requirement is also a string:
```
template_fields = ("requirement",)
template_ext = (".txt",)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847735311
How about this (I know it's not perfect, but I think it should do the job):
```
requirements = (lambda: open("requirements.txt").readlines())()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847999179
> I think in your use case the whole requirement file is in fact not a requirement.txt but a value pushed to xcom by previous task.
No. I am talking about "requirement.txt" file in which some values might be provided by XCom.
I can imagine this kind of requirement.txt file for example:
```
dependency1==1.0.0
dependency2==2.0.0
dependency3=={{ code to retrieve depedency version from xcom}}
dependency4==1.1.0
{{- % if retrireve deployment == 'production' % }}
# only needed in production deployment
dependency5==5.0.0
{{- % endif % }}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847744371
Agree. Having a good, templated parameter could work. Now when I think obout it might actually be as simple as adding this to the operator and handling the case where `requirements` is also a string and properly using it with `-r`/temp file by the operator in this case:
```
template_fields = ("requirement",)
template_ext = (".txt",)
```
The `template_ext` is one of the lesser-known features of Airflow operators
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847782934
Or maybe we can do something similar to `bash_command`? If a `requirements` argument looks like a file (`os.path.isfile()` or contains `os.sep`), automatically insert a `-r` in from of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847730376
Can you see if you can pass `["-r", "requireements.txt]` as. requirements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexInhert commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
alexInhert commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847805535
But in this case you want the functionality of `template_ext` without the `template_fields`
There is no need to go through jinja engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847744371
Agree. Having a good, templated parameter could work. Now when I think obout it might actually be as simple as adding this to the operator and handling the case where `requirements` is also a string and properly using it with `-r`/temp file by the operator in this case:
```
template_fields = ("requirements",)
template_ext = (".txt",)
```
The `template_ext` is one of the lesser-known features of Airflow operators
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847807490
> But in this case you want the functionality of `template_ext` without the `template_fields`
> There is no need to go through jinja engine
But this might be very useful. For example you can use jinja variable for version of a dependency and get it from another task. or have conditional requirements.txt content based on dev/prod settings etc.
This is pretty useful feature IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847807490
> But in this case you want the functionality of `template_ext` without the `template_fields`
> There is no need to go through jinja engine
But this might be very useful. For example you can use jinja variale for version of a dependency and get it from another task. or have conditional requirements.txt content based on dev/prod settings etc.
This is pretty useful feature IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847744371
Agree. Having a good, templated parameter could work. Now when I think obout it might actually be as simple as adding this to the operator and handling the case where `requirements` is also a string and properly using it with `-r`/temp file by the operator in this case:
```
template_fields = ("requirement",)
template_ext = (".txt",)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16037: allow using requirments.txt in PythonVirtualEnvOperator
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16037:
URL: https://github.com/apache/airflow/issues/16037#issuecomment-847797452
I thinkl it the same feature we are talking about :).
The "template_ext" works in the way that it will read the file as string when the parameter has the extension registered there and it will automatically process the file trough the template engine, so you can add jinja-variables from context /xcom variables in the file. It's built-in feature in BaseOperator
All that you need is in this case to have the right extension (".txt") in the value of the parameter.
```
requirements="[<path>/]reqiurements.txt"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org