You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/12 20:37:27 UTC

[GitHub] [airflow] vinny-santaiti opened a new issue, #25008: airflow tasks test cli not seeing xcom result

vinny-santaiti opened a new issue, #25008:
URL: https://github.com/apache/airflow/issues/25008

   ### Apache Airflow version
   
   2.3.3 (latest released)
   
   ### What happened
   
   The `airflow tasks test` CLI doesn't seem to work for xcoms, it used to work in version 2.2.5. It works fine in the UI.
   Example: airflow tasks test tutorial print_date 2015-06-01
   ```kwargs['ti'].xcom_pull(task_ids='push_data')```
   https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html#id2
   When you push data in a task, then pull xcom data by task_ids in a downstream task, it doesnt work.
   
   ### What you think should happen instead
   
   The same cli cmd did work fine in airflow 2.2.5
   
   ### How to reproduce
   
   ```python
   import pendulum
   
   from airflow import DAG
   from airflow.operators.python import PythonOperator
   
   
   with DAG(
       'example_xcom',
       schedule_interval=None,
       start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
       catchup=False,
   ) as dag:
   
       def push_data(**kwargs):
           return 'data'
   
       def pull_data(**kwargs):
           return kwargs['ti'].xcom_pull(task_ids='push_data')
   
       push_task = PythonOperator(
           task_id='push_data',
           python_callable=push_data
       )
       pull_task = PythonOperator(
           task_id='pull_data',
           python_callable=pull_data
       )
       push_task >> pull_task
   ```
   ```
   airflow tasks test example_xcom push_data 2022-06-01
   [2022-07-12 20:22:53,288] {python.py:173} INFO - Done. Returned value was: data
   airflow tasks test example_xcom pull_data 2022-06-01
   [2022-07-12 20:23:05,402] {python.py:173} INFO - Done. Returned value was: None
   ```
   
   ### Operating System
   
   docker
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow==2.3.3
   apache-airflow-providers-amazon==4.0.0
   apache-airflow-providers-celery==3.0.0
   apache-airflow-providers-cncf-kubernetes==4.1.0
   apache-airflow-providers-docker==3.0.0
   apache-airflow-providers-elasticsearch==4.0.0
   apache-airflow-providers-ftp==3.0.0
   apache-airflow-providers-google==8.1.0
   apache-airflow-providers-grpc==3.0.0
   apache-airflow-providers-hashicorp==3.0.0
   apache-airflow-providers-http==3.0.0
   apache-airflow-providers-imap==3.0.0
   apache-airflow-providers-microsoft-azure==4.0.0
   apache-airflow-providers-mysql==3.0.0
   apache-airflow-providers-odbc==3.0.0
   apache-airflow-providers-postgres==5.0.0
   apache-airflow-providers-redis==3.0.0
   apache-airflow-providers-sendgrid==3.0.0
   apache-airflow-providers-sftp==3.0.0
   apache-airflow-providers-slack==5.0.0
   apache-airflow-providers-sqlite==3.0.0
   apache-airflow-providers-ssh==3.0.0
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   apache/airflow:2.3.3-python3.8
   https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed issue #25008: airflow tasks test cli not seeing xcom result

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25008: airflow tasks test cli not seeing xcom result
URL: https://github.com/apache/airflow/issues/25008


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #25008: airflow tasks test cli not seeing xcom result

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #25008:
URL: https://github.com/apache/airflow/issues/25008#issuecomment-1484256929

   > That is pretty sad. We used to relay in our test suit on combo of `airflow tasks test` and XCom.set and XCom.get_one. After upgrade, we have to refactor a lot. BTW any chance to combine now XCom.set and relay on it in task executes self.xcom_pull meanwhile using `airflow tasks test` for testing?
   
   Sure. PRs are always welcome :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on issue #25008: airflow tasks test cli not seeing xcom result

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #25008:
URL: https://github.com/apache/airflow/issues/25008#issuecomment-1182605697

   It’s less unintended, more that we considered this as a bug and not actually relied on by users :(
   
   It’s viable to add a flag for this, although it should communicated quite clearly that this will leave junk in the database and you will need to be fine with that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] andilabs commented on issue #25008: airflow tasks test cli not seeing xcom result

Posted by "andilabs (via GitHub)" <gi...@apache.org>.
andilabs commented on issue #25008:
URL: https://github.com/apache/airflow/issues/25008#issuecomment-1482961499

   That is pretty sad. We used to relay in our test suit on combo of `airflow tasks test` and XCom.set and XCom.get_one. After upgrade, we have to refactor a lot. BTW any chance to combine now XCom.set and relay on it in task executes self.xcom_pull meanwhile using  `airflow tasks test`  for testing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #25008: airflow tasks test cli not seeing xcom result

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25008:
URL: https://github.com/apache/airflow/issues/25008#issuecomment-1182587335

   Ths is not a bug. Simply previous behaviour was an unintended side effect.
   
   via https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html#id2
   
   > Note that the airflow tasks test command runs task instances locally, outputs their log to stdout (on screen), does not bother with dependencies, and does not communicate state (running, success, failed, …) to the database. It simply allows testing a single task instance.
   
   While "Xcom" is not specified here, leaving an XCom behing from `test` command is something that falls into that .
   
   In 2.3. Foreign Key was added between XCom and Task Instance - https://github.com/apache/airflow/pull/22334 (it was missing before). The result of missing foreign key was that the XCom created during the 'test" command was never cleaned. 
   
   This piece of code https://github.com/apache/airflow/commit/71c980a8ffb3563bf16d8a23a58de54c9e8cf556#diff-ad618185a072910e49c11770954af009d1cc070b120a4fde5f2fc095a588271bR550 removes the dag run (and thus transitively task instance and xcom) created during the tests)
   
   I believe (@uranusjr ? Can you confirm please) this was fully intentional change.
    
   Maybe this is a good case however to add the version of `airflow tasks test` (`aiflow tasks tests --no-cleanup` ?) command that actually leaves some traces behind. WDYT @uranusjr ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #25008: airflow tasks test cli not seeing xcom result

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25008:
URL: https://github.com/apache/airflow/issues/25008#issuecomment-1182929003

   So as I thought. Closing as invalid. @vinny-santaiti - if you find it useful to have similar feature with leaving data after tests - feel free to make PR to add such feature. Generally speaking "tasks test" is only intended to run each task in isolation and not leave traces.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org