You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/10 14:49:35 UTC
[GitHub] [airflow] oriolcmp opened a new issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
oriolcmp opened a new issue #12985:
URL: https://github.com/apache/airflow/issues/12985
Apache Airflow version: 1.10.13
Environment: Docker (Ubuntu 18.4 - Python 3.7)
Cloud provider or hardware configuration: Azure
OS (e.g. from /etc/os-release): Docker (Ubuntu 18.4 - Python 3.7)
Kernel (e.g. uname -a): Docker (Ubuntu 18.4 - Python 3.7)
Install tools: N/A
Others: N/A
What happened:
When we enable provide_context=True for PythonVirtualenvOperator and try to use the xcom_push to pass a variable I get this error:
`File "/tmp/venv0upgqome/script.py", line 13, in push\n kwargs[\'ti\'].xcom_push(key=\'value from pusher 1\', value=value_1)\nKeyError: \'ti\'\n'`
**How to reproduce it:**
```
import airflow
from airflow import DAG
from airflow.operators.python_operator import PythonVirtualenvOperator
args = {
'owner': 'Airflow',
'start_date': airflow.utils.dates.days_ago(2),
}
dag = DAG('BAtatas', schedule_interval="@once", default_args=args)
def push(**kwargs):
"""Pushes an XCom without a specific target"""
value_1 = [1, 2, 3]
print("printing the kwargs!!!")
print(kwargs)
kwargs['ti'].xcom_push(key='value from pusher 1', value=value_1)
def push_by_returning(**kwargs):
value_2 = {'a': 'b'}
"""Pushes an XCom without a specific target, just by returning it"""
return value_2
def puller(**kwargs):
value_2 = {'a': 'b'}
value_1 = [1, 2, 3]
"""Pull all previously pushed XComs and check if the pushed values match the pulled values."""
ti = kwargs['ti']
# get value_1
pulled_value_1 = ti.xcom_pull(key=None, task_ids='push')
assert pulled_value_1 == value_1
# get value_2
pulled_value_2 = ti.xcom_pull(task_ids='push_by_returning')
assert pulled_value_2 == value_2
# get both value_1 and value_2
pulled_value_1, pulled_value_2 = ti.xcom_pull(
key=None, task_ids=['push', 'push_by_returning'])
assert (pulled_value_1, pulled_value_2) == (value_1, value_2)
push1 = PythonVirtualenvOperator(
task_id='push',
dag=dag,
python_callable=push,
requirements=[],
python_version='3.7',
use_dill=False,
provide_context=True,
system_site_packages=True,
)
push2 = PythonVirtualenvOperator(
task_id='push_by_returning',
dag=dag,
python_callable=push_by_returning,
requirements=[],
python_version='3.7',
use_dill=False,
provide_context=True,
system_site_packages=True,
)
pull = PythonVirtualenvOperator(
task_id='puller',
dag=dag,
python_callable=puller,
requirements=[],
python_version='3.7',
use_dill=False,
provide_context=True,
system_site_packages=True,
)
pull << [push1, push2]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
RosterIn commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-818515506
I thought provide_context was removed from Python operators in Airflow 2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xinbinhuang commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-830928107
> Also found this Jira story (https://issues.apache.org/jira/browse/AIRFLOW-2738) that's still open that seems to be addressing the same problem.
Just a quick note: the community has migrated issues tracking system from Jira to GitHub issues, so whatever left in Jira right now is likely outdated or inaccurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-818603744
It was, and what `provide_context` provides is now generally always available. So this issue (in 2.0) is more about there is no way to expose `ti` to the callable run by a `PythonVirtualenvOperator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dsynkov commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
dsynkov commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-818158798
Sharing a few notes as I've run into this same issue:
* The [docs](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator) for `PythonVirtualenvOperator` address this:
> You can use the op_args and op_kwargs arguments the same way you use it in the PythonOperator. Unfortunately we currently do not support to serialize var and ti / task_instance **due to incompatibilities with the underlying library**.
* I'm not sure what the exact "incompatibilities" are, but searching for open issues mentioning `PythonVirtualenvOperator` there is https://github.com/apache/airflow/issues/7870 which addresses using `dill` vs. `cloudpickle` (but doesn't mention how/if this would enable serialization of `ti`, etc.). Anyone have any insight if https://github.com/apache/airflow/issues/7870 is a pre-req to this current issue?
* Also found this Jira story (https://issues.apache.org/jira/browse/AIRFLOW-2738) that's still open that seems to be addressing the same problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xinbinhuang edited a comment on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
xinbinhuang edited a comment on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-830928107
> Also found this Jira story (https://issues.apache.org/jira/browse/AIRFLOW-2738) that's still open that seems to be addressing the same problem.
Just a quick note: the community has migrated the issues tracking system from Jira to GitHub issues, so whatever left in Jira right now is likely outdated or inaccurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] oriolcmp commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
oriolcmp commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-802891139
Hello,
Do you have any news on this issue? It would be very useful for us to use it.
Thanks,
oriol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-812762009
I have un-assigned myself so it is up for grabs :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-869594775
Sound like a feature to add rather than bug-fix I am moving it to 2.2 milestone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] oriolcmp commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
oriolcmp commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-944295056
Sorry but I don't think it shouldn't be a feature. It is a bug because If I'm not wrong all operators should be able to use XCom variables. And with this operator currently this is not possible. This operator could be very useful to deal with dependency contraints but without passing variables is kind of limited and it seems the only option left is to use DockerOperator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #12985: PythonVirtualenvOperator with provide_context=True does not have 'ti' keyword
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12985:
URL: https://github.com/apache/airflow/issues/12985#issuecomment-742567771
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org