You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "WiC-htao (via GitHub)" <gi...@apache.org> on 2025/01/17 06:06:56 UTC
[I] Compatibility issues with polars: Any calculation of polars out of a task will block calculation in the task [airflow]
WiC-htao opened a new issue, #45730:
URL: https://github.com/apache/airflow/issues/45730
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.10.0
### What happened?
When I create a constant out of tasks using some polars calculation like df.unique or df.with_columns, any polars calculation in tasks will be stuck. The tasks including it will show running and never end.
my polars version is 0.20.19
### What you think should happen instead?
The task should run as expected
### How to reproduce
```
from airflow.decorators import task
from airflow.models.dag import DAG
import polars as pl
import pendulum
S1_TS = pl.DataFrame({"ticktime": [1, 2, 2, 3]}).unique()
with DAG(
f"test",
schedule="00 20 * * 1-5",
start_date=pendulum.datetime(2024, 6, 18),
) as dag:
# S1_TS = pl.DataFrame({"ticktime": [1, 2, 2, 3]}).unique() # created constant in the dag context will have the same stucks
@task
def test():
x = pl.DataFrame({"a": [1, 2]}).filter(a=1)
test()
```
### Operating System
Ubuntu 20.04.6 LTS (Focal Fossa)
### Versions of Apache Airflow Providers

### Deployment
Official Apache Airflow Helm Chart
### Deployment details
apache-airflow-providers-cncf-kubernetes 8.3.0
apache-airflow-providers-common-compat 1.2.0
apache-airflow-providers-common-io 1.3.1
apache-airflow-providers-common-sql 1.13.0
apache-airflow-providers-fab 1.1.0
apache-airflow-providers-ftp 3.9.0
apache-airflow-providers-http 4.11.0
apache-airflow-providers-imap 3.6.0
apache-airflow-providers-mysql 5.6.1
apache-airflow-providers-smtp 1.7.0
apache-airflow-providers-sqlite 3.8.0
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Compatibility issues with polars: Any calculation of polars out of a task will block calculation in the task [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #45730: Compatibility issues with polars: Any calculation of polars out of a task will block calculation in the task
URL: https://github.com/apache/airflow/issues/45730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org