You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "IH-Miller (via GitHub)" <gi...@apache.org> on 2025/06/17 13:41:40 UTC
[D] New Dags not loading [airflow]
GitHub user IH-Miller created a discussion: New Dags not loading
Using
-Airflow version 2.8.4
-Python version 3.11.10
Order of events:
1.In Nov 2024 we upgraded Airflow python from 3.8 -> 3.11
A.This forced apache-airflow-providers-snowflake to then use pandas 2.1.4 instead of 2.0.3 because of the requirement of ">=2.1.2"
B.apache-airflow 2.8.4 says it can handle <2.2
C.While testing existing DAGS seems to work and adding new DAGS worked
2.In April 2025 we did a security vunerability upgrade of the Snowflake connector
A.Upgrade snowflake-connector-python from 3.7.1 to 3.14.0
B.Testing DAGS worked and we had no errors. However we did not test adding new DAGS at this time.
3.2025-06-13: We tried to adding multiple new DAGS to the airflow server and it wasn't showing in the UI with no errors displayed either
A."airflow DAGS list" shows the new dag with the paused field showing "None" instead of True/False
B."airflow DAGS list-import-errors" shows no errors for the new DAG
C.We tried running "airflow DAGS unpause <dag_name>" and it gave an error saying the DAG did not exist on the table dag.
D.We tried running existing DAGS an they all ran correctly had no error and on "airflow DAGS list" have a Paused value of true or false
E.We tried creating empty DAGS and "airflow DAGS list" shows the new DAG with the paused field showing "None" instead of True/False
F.We downloaded dagbag(Re-serialize any DAGS missing from the database.) python file from the internet
i.We ran "airflow DAGS list" again it shows less tasks those not using the snowflake-connector-python library and all the DAGs using snowflake-connector-python
showed errors even those that were old and still worked.
ii.Then show errors in the "airflow DAGS list-import-errors" output
Output:
a.File "hashtable.pyx", line 1, in init pandas._libs.hashtable
b.File "missing.pyx", line 42, in init pandas._libs.missing
c.AttributeError: partially initialized module 'pandas' has no attribute '_pandas_datetime_CAPI' (most likely due to a circular import)
iii.We tried using pandas==2.1.2 and 2.1.4 this didn't resolve the issue
iv.We tried reverting snowflake-connector-python and pandas back to the versions of libraries we had in November 2024 and we still had the issue.
[requirements.txt](https://github.com/user-attachments/files/20776472/requirements.txt)
GitHub link: https://github.com/apache/airflow/discussions/51841
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] New Dags not loading [airflow]
Posted by "IH-Miller (via GitHub)" <gi...@apache.org>.
GitHub user IH-Miller closed the discussion with a comment: New Dags not loading
Error Above masked the real issue. Real issue was found in the schedular logs and pointed towards an issue in the database. Once we resolved the database issue the issue was resolved for adding new DAGs
GitHub link: https://github.com/apache/airflow/discussions/51841#discussioncomment-13507935
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] New Dags not loading [airflow]
Posted by "IH-Miller (via GitHub)" <gi...@apache.org>.
GitHub user IH-Miller edited a discussion: New Dags not loading
Using
-Airflow version 2.8.4
-Python version 3.11.10
Order of events:
1.In Nov 2024 we upgraded Airflow python from 3.8 -> 3.11
A.This forced apache-airflow-providers-snowflake to then use pandas 2.1.4 instead of 2.0.3 because of the requirement of ">=2.1.2"
B.apache-airflow 2.8.4 says it can handle <2.2
C.While testing existing DAGS seems to work and adding new DAGS worked
2.In April 2025 we did a security vunerability upgrade of the Snowflake connector
A.Upgrade snowflake-connector-python from 3.7.1 to 3.14.0
B.Testing DAGS worked and we had no errors. However we did not test adding new DAGS at this time.
3.2025-06-13: We tried to adding multiple new DAGS to the airflow server and it wasn't showing in the UI with no errors displayed either
A."airflow DAGS list" shows the new dag with the paused field showing "None" instead of True/False
B."airflow DAGS list-import-errors" shows no errors for the new DAG
C.We tried running "airflow DAGS unpause <dag_name>" and it gave an error saying the DAG did not exist on the table dag.
D.We tried running existing DAGS an they all ran correctly had no error and on "airflow DAGS list" have a Paused value of true or false
E.We tried creating empty DAGS and "airflow DAGS list" shows the new DAG with the paused field showing "None" instead of True/False
F.We downloaded dagbag(Re-serialize any DAGS missing from the database.) python file from the internet
i.We ran "airflow DAGS list" again it shows less tasks those not using the snowflake-connector-python library and all the DAGs using snowflake-connector-python
showed errors even those that were old and still worked.
ii.Then show errors in the "airflow DAGS list-import-errors" output
Output:
a.File "hashtable.pyx", line 1, in init pandas._libs.hashtable
b.File "missing.pyx", line 42, in init pandas._libs.missing
c.AttributeError: partially initialized module 'pandas' has no attribute '_pandas_datetime_CAPI' (most likely due to a circular import)
iii.We tried using pandas==2.1.2 and 2.1.4 this didn't resolve the issue
iv.We tried reverting snowflake-connector-python and pandas back to the versions of libraries we had in November 2024 and we still had the issue.
[requirements.txt](https://github.com/user-attachments/files/20776472/requirements.txt)
GitHub link: https://github.com/apache/airflow/discussions/51841
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] New Dags not loading [airflow]
Posted by "IH-Miller (via GitHub)" <gi...@apache.org>.
GitHub user IH-Miller closed a discussion: New Dags not loading
Using
-Airflow version 2.8.4
-Python version 3.11.10
Order of events:
1.In Nov 2024 we upgraded Airflow python from 3.8 -> 3.11
A.This forced apache-airflow-providers-snowflake to then use pandas 2.1.4 instead of 2.0.3 because of the requirement of ">=2.1.2"
B.apache-airflow 2.8.4 says it can handle <2.2
C.While testing existing DAGS seems to work and adding new DAGS worked
2.In April 2025 we did a security vunerability upgrade of the Snowflake connector
A.Upgrade snowflake-connector-python from 3.7.1 to 3.14.0
B.Testing DAGS worked and we had no errors. However we did not test adding new DAGS at this time.
3.2025-06-13: We tried to adding multiple new DAGS to the airflow server and it wasn't showing in the UI with no errors displayed either
A."airflow DAGS list" shows the new dag with the paused field showing "None" instead of True/False
B."airflow DAGS list-import-errors" shows no errors for the new DAG
C.We tried running "airflow DAGS unpause <dag_name>" and it gave an error saying the DAG did not exist on the table dag.
D.We tried running existing DAGS an they all ran correctly had no error and on "airflow DAGS list" have a Paused value of true or false
E.We tried creating empty DAGS and "airflow DAGS list" shows the new DAG with the paused field showing "None" instead of True/False
F.We downloaded dagbag(Re-serialize any DAGS missing from the database.) python file from the internet
i.We ran "airflow DAGS list" again it shows less tasks those not using the snowflake-connector-python library and all the DAGs using snowflake-connector-python
showed errors even those that were old and still worked.
ii.Then show errors in the "airflow DAGS list-import-errors" output
Output:
a.File "hashtable.pyx", line 1, in init pandas._libs.hashtable
b.File "missing.pyx", line 42, in init pandas._libs.missing
c.AttributeError: partially initialized module 'pandas' has no attribute '_pandas_datetime_CAPI' (most likely due to a circular import)
iii.We tried using pandas==2.1.2 and 2.1.4 this didn't resolve the issue
iv.We tried reverting snowflake-connector-python and pandas back to the versions of libraries we had in November 2024 and we still had the issue.
[requirements.txt](https://github.com/user-attachments/files/20776472/requirements.txt)
GitHub link: https://github.com/apache/airflow/discussions/51841
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] New Dags not loading [airflow]
Posted by "IH-Miller (via GitHub)" <gi...@apache.org>.
GitHub user IH-Miller edited a discussion: New Dags not loading
Using
-Airflow version 2.8.4
-Python version 3.11.10
Order of events:
1.In Nov 2024 we upgraded Airflow python from 3.8 -> 3.11
A.This forced apache-airflow-providers-snowflake to then use pandas 2.1.4 instead of 2.0.3 because of the requirement of ">=2.1.2"
B.apache-airflow 2.8.4 says it can handle <2.2
C.While testing existing DAGS seems to work and adding new DAGS worked
2.In April 2025 we did a security vunerability upgrade of the Snowflake connector
A.Upgrade snowflake-connector-python from 3.7.1 to 3.14.0
B.Testing DAGS worked and we had no errors. However we did not test adding new DAGS at this time.
3.2025-06-13: We tried to adding multiple new DAGS to the airflow server and it wasn't showing in the UI with no errors displayed either
A."airflow DAGS list" shows the new dag with the paused field showing "None" instead of True/False
B."airflow DAGS list-import-errors" shows no errors for the new DAG
C.We tried running "airflow DAGS unpause <dag_name>" and it gave an error saying the DAG did not exist on the table dag.
D.We tried running existing DAGS an they all ran correctly had no error and on "airflow DAGS list" have a Paused value of true or false
E.We tried creating empty DAGS and "airflow DAGS list" shows the new DAG with the paused field showing "None" instead of True/False
F.We downloaded dagbag(Re-serialize any DAGS missing from the database.) python file from the internet
i.We ran "airflow DAGS list" again it shows less tasks those not using the snowflake-connector-python library and all the DAGs using snowflake-connector-python
showed errors even those that were old and still worked.
ii.Then show errors in the "airflow DAGS list-import-errors" output
Output:
a.File "hashtable.pyx", line 1, in init pandas._libs.hashtable
b.File "missing.pyx", line 42, in init pandas._libs.missing
c.AttributeError: partially initialized module 'pandas' has no attribute '_pandas_datetime_CAPI' (most likely due to a circular import)
iii.We tried using pandas==2.1.2 and 2.1.4 this didn't resolve the issue
iv.We tried reverting snowflake-connector-python and pandas back to the versions of libraries we had in November 2024 and we still had the issue.
[requirements.txt](https://github.com/user-attachments/files/20776472/requirements.txt)
GitHub link: https://github.com/apache/airflow/discussions/51841
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] New Dags not loading [airflow]
Posted by "IH-Miller (via GitHub)" <gi...@apache.org>.
GitHub user IH-Miller edited a discussion: New Dags not loading
`Using
-Airflow version 2.8.4
-Python version 3.11.10
Order of events:
1.In Nov 2024 we upgraded Airflow python from 3.8 -> 3.11
A.This forced apache-airflow-providers-snowflake to then use pandas 2.1.4 instead of 2.0.3 because of the requirement of ">=2.1.2"
B.apache-airflow 2.8.4 says it can handle <2.2
C.While testing existing DAGS seems to work and adding new DAGS worked
2.In April 2025 we did a security vunerability upgrade of the Snowflake connector
A.Upgrade snowflake-connector-python from 3.7.1 to 3.14.0
B.Testing DAGS worked and we had no errors. However we did not test adding new DAGS at this time.
3.2025-06-13: We tried to adding multiple new DAGS to the airflow server and it wasn't showing in the UI with no errors displayed either
A."airflow DAGS list" shows the new dag with the paused field showing "None" instead of True/False
B."airflow DAGS list-import-errors" shows no errors for the new DAG
C.We tried running "airflow DAGS unpause <dag_name>" and it gave an error saying the DAG did not exist on the table dag.
D.We tried running existing DAGS an they all ran correctly had no error and on "airflow DAGS list" have a Paused value of true or false
E.We tried creating empty DAGS and "airflow DAGS list" shows the new DAG with the paused field showing "None" instead of True/False
F.We downloaded dagbag(Re-serialize any DAGS missing from the database.) python file from the internet
i.We ran "airflow DAGS list" again it shows less tasks those not using the snowflake-connector-python library and all the DAGs using snowflake-connector-python
showed errors even those that were old and still worked.
ii.Then show errors in the "airflow DAGS list-import-errors" output
Output:
a.File "hashtable.pyx", line 1, in init pandas._libs.hashtable
b.File "missing.pyx", line 42, in init pandas._libs.missing
c.AttributeError: partially initialized module 'pandas' has no attribute '_pandas_datetime_CAPI' (most likely due to a circular import)
iii.We tried using pandas==2.1.2 and 2.1.4 this didn't resolve the issue
iv.We tried reverting snowflake-connector-python and pandas back to the versions of libraries we had in November 2024 and we still had the issue.
[requirements.txt](https://github.com/user-attachments/files/20776472/requirements.txt)`
GitHub link: https://github.com/apache/airflow/discussions/51841
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org