You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "WiC-htao (via GitHub)" <gi...@apache.org> on 2025/01/17 06:06:56 UTC

[I] Compatibility issues with polars: Any calculation of polars out of a task will block calculation in the task [airflow]

WiC-htao opened a new issue, #45730:
URL: https://github.com/apache/airflow/issues/45730

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.10.0
   
   ### What happened?
   
   When I create a constant out of tasks using some polars calculation like df.unique or df.with_columns, any polars calculation in tasks will be stuck. The tasks including it will show running and never end.
   
   my polars version is 0.20.19
   
   ### What you think should happen instead?
   
   The task should run as expected
   
   ### How to reproduce
   
   ```
   from airflow.decorators import task
   from airflow.models.dag import DAG
   import polars as pl
   import pendulum    
   S1_TS = pl.DataFrame({"ticktime": [1, 2, 2, 3]}).unique()
   
   with DAG(
       f"test",
       schedule="00 20 * * 1-5",
       start_date=pendulum.datetime(2024, 6, 18),
   ) as dag:
       # S1_TS = pl.DataFrame({"ticktime": [1, 2, 2, 3]}).unique() # created constant in the dag context will have the same stucks
       @task
       def test():
           x = pl.DataFrame({"a": [1, 2]}).filter(a=1)
       test()
   ```
   
   ### Operating System
   
   Ubuntu 20.04.6 LTS (Focal Fossa)
   
   ### Versions of Apache Airflow Providers
   
   ![Image](https://github.com/user-attachments/assets/70cb7db2-3a1b-42bf-8ebe-c73e2d9075d2)
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   apache-airflow-providers-cncf-kubernetes 8.3.0
   apache-airflow-providers-common-compat   1.2.0
   apache-airflow-providers-common-io       1.3.1
   apache-airflow-providers-common-sql      1.13.0
   apache-airflow-providers-fab             1.1.0
   apache-airflow-providers-ftp             3.9.0
   apache-airflow-providers-http            4.11.0
   apache-airflow-providers-imap            3.6.0
   apache-airflow-providers-mysql           5.6.1
   apache-airflow-providers-smtp            1.7.0
   apache-airflow-providers-sqlite          3.8.0
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Compatibility issues with polars: Any calculation of polars out of a task will block calculation in the task [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #45730: Compatibility issues with polars: Any calculation of polars out of a task will block calculation in the task
URL: https://github.com/apache/airflow/issues/45730


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org