You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "TJaniF (via GitHub)" <gi...@apache.org> on 2025/03/30 18:20:18 UTC

[I] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` when the Variable is used at the top level of the dag [airflow]

TJaniF opened a new issue, #48554:
URL: https://github.com/apache/airflow/issues/48554

   ### Apache Airflow version
   
   3.0.0
   
   ### If "Other Airflow 2 version" selected, which one?
   
   main
   
   ### What happened?
   
   This dag serializes and runs fine for `from airflow.models import Variable` but not for `from airflow.sdk import Variable`
   
   ```
   from airflow.sdk import dag, Variable
   from airflow.decorators import task 
   # from airflow.models import Variable
   
   # This one only works for imports from airflow.models, not from airflow.sdk
   var1 = Variable.get("v1", "TOP_LEVEL_VAR")
   
   @dag()
   def test_var():
   
       @task 
       def get_var():
           print("TOP LEVEL VAR ", var1)
   
           # this one works for both imports, from airflow.sdk and from airflow.models
           var2 = Variable.get("v2", "TASK_LEVEL_VAR")
           print("TASK LEVEL VAR ", var2)
       
       get_var() 
   
   
   test_var()
   ```
   
   The error is only caused when the Variable is used at the top level of the dag, not when within a task. 
   
   Error:
   
   ```
   Traceback (most recent call last):
     File "/opt/airflow/task-sdk/src/airflow/sdk/definitions/variable.py", line 53, in get
       return _get_variable(key, deserialize_json=deserialize_json)
     File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/context.py", line 185, in _get_variable
       from airflow.sdk.execution_time.task_runner import SUPERVISOR_COMMS
   ImportError: cannot import name 'SUPERVISOR_COMMS' from 'airflow.sdk.execution_time.task_runner' (/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py)
   ```
   
   ### What you think should happen instead?
   
   The dag should serialize and run with the updated import.
   
   ### How to reproduce
   
   Add the above dag to your env, run airflow dags reserialize, see the error. Switch the import of Variable to `airflow.models` to see the dag work.
   
   ### Operating System
   
   MacOS
   
   ### Versions of Apache Airflow Providers
   
   None
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   breeze
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "fromm1990 (via GitHub)" <gi...@apache.org>.
fromm1990 commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2903910966

   I seem to get this error when using `Variable.get("my_variable")` inside an external_python task. The variable is perfectly able to be retrieved from an ordinary task. The variable is also listed when executing the `airflow variables list` command from an python environment activated terminal. The latter suggests, that the virtual environment should be properly set up, in my opinion. This setup worked fine in airflow 2.10.5 (however using `from airflow.models import Variable` instead of `from airflow.sdk import Variable`)
   
   I don't know if the cause of this issue is the same, but sounds similar to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "alkismavridis (via GitHub)" <gi...@apache.org>.
alkismavridis commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-3004298029

   Any news one that? We heavily use `airflow dags reserialze` and this error has significantly affected our workflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2780749685

   > I am able to reproduce it locally with main development branch. However, having some difficulty in understanding how to fix this. As also mentioned above: SUPERVISOR_COMMS is getting initialized only inside the @task get_var method. Any recommendations on how to get started ? Thanks
   
   Thanks for checking! I will check and add some pointers soon. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "dshvedchenko (via GitHub)" <gi...@apache.org>.
dshvedchenko commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-3183278358

   faced this issue in tests, when forget to update mock to  `airflow.sdk.Variables.get`  from `airflow.models.Variables.get`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` when the Variable is used at the top level of the dag [airflow]

Posted by "TJaniF (via GitHub)" <gi...@apache.org>.
TJaniF commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2765653799

   Tried on the latest main and the issue seems to be limited to the `airflow dags reseialize` command. The dag processor can handle the dag fine. So not the most high priority issue in this case :) 
   
   The screenshot below shows the cli command output with the error and having processed the other 3 dags in my env, while the dag processor at the same time is happy to process all 4 dags including the one in this issue
   
   <img width="1162" alt="Image" src="https://github.com/user-attachments/assets/ee16cba4-ba57-483b-ae3c-1e0b8d83448e" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "Prajwal7842 (via GitHub)" <gi...@apache.org>.
Prajwal7842 commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2780740767

   I am able to reproduce it locally with main development branch. However, having some difficulty in understanding how to fix this. As also mentioned above: SUPERVISOR_COMMS is getting initialized only inside the @task get_var method. Any recommendations on how to get started ? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "Prajwal7842 (via GitHub)" <gi...@apache.org>.
Prajwal7842 commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-3418442378

   I think the original issue in the thread is fixed. I am unable to reproduce this on latest main with airflow dags reserialize command. SS attached. 
   [
   
   <img width="1725" height="589" alt="Image" src="https://github.com/user-attachments/assets/686beb33-d5d9-49f9-81ea-d52af5de7ba7" />
   
   ](url)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "farkmarnum (via GitHub)" <gi...@apache.org>.
farkmarnum commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2949764357

   We're seeing this same error when running tests:
   ```py
   # Passes
   def test_with_models_import() -> None:
       from airflow.models import Variable
   
       Variable.set("foo", "bar")
       assert Variable.get("foo") == "bar"
   
   
   # Fails
   def test_with_sdk_import() -> None:
       from airflow.sdk import Variable
   
       Variable.set("foo", "bar")
       assert Variable.get("foo") == "bar"
   ```
   
   The seconds test fails with:
   ```log
   FAILED tests/dags/temp_test.py::test_with_sdk_import - ImportError:
   cannot import name 'SUPERVISOR_COMMS' from 'airflow.sdk.execution_time.task_runner'
   (<conda env dir>/lib/python3.12/site-packages/airflow/sdk/execution_time/...
   ```
   
   Both on Airflow 3.0.1 and 3.0.2rc2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-6261876500

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "jablecherman (via GitHub)" <gi...@apache.org>.
jablecherman commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2919518648

   > I faced the same ImportError trying to evaluate `LazyXComSequence(<unevaluated length>)` in the reduce function during dynamic task mapping. Running with `dag.test()`, LocalExecutor, Airflow 3.0.1.
   > 
   > # Reduce function
   > @task
   > def send_notifications(data_to_notificate: list[tuple[Rule, ObjectStoragePath]], **context) -> None:
   >     for rule, input_file in data_to_notificate: # The error raises here
   >         ...
   > UPD: the same parameter is identified as `LazySelectSequence([4 items])` in Airflow 2.11.0, and the DAG works properly.
   
   
   Seconding the `dag.test()` note (apologies for non-task flow syntax)
   
   ```
   from airflow.models.dag import DAG
   from airflow.providers.standard.operators.python import PythonOperator
   from airflow.sdk.definitions.connection import Connection
   
   MSSQL_CONN_ID = "test_conn_id"
   
   with DAG(
       dag_id="test_dag",
       catchup=False,
   ) as dag:
       PythonOperator(
           task_id = "foo",
           python_callable=Connection.get,
           op_args=[MSSQL_CONN_ID]
       )
   
   if __name__ == "__main__":
       dag.test()
   
   ```
   
   Running this DAG via dag.test() results in this error, running normally (via UI) is fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "Dev-iL (via GitHub)" <gi...@apache.org>.
Dev-iL commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2897444411

   I encountered this issue on 3.0.1 when running a test that creates a dagbag. It went away after I added this to pytest `filterwarnings`:
   
   ```toml
   # pyproject.toml
   [tool.pytest.ini_options]
   filterwarnings = [
       "ignore:Timer and timing metrics publish in seconds were deprecated.*:airflow.exceptions.RemovedInAirflow4Warning:airflow",
   ]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "Prajwal7842 (via GitHub)" <gi...@apache.org>.
Prajwal7842 commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2768113144

   Hi @vatsrahul1001 Since this is marked as low priority. Can I try this out and propose a fix, if this is still an issue in latest main ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` when the Variable is used at the top level of the dag [airflow]

Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2765180108

   @TJaniF I tried it on main and it works well. Can you try on latest main ?
   
   ![Image](https://github.com/user-attachments/assets/00569ea1-c39b-47e3-bb4f-b3ee39e0d65a)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2769728195

   > Hi [@vatsrahul1001](https://github.com/vatsrahul1001) Since this is marked as low priority. Can I try this out and propose a fix, if this is still an issue in latest main ?
   
   You can start from reproducing it and checking it - I don't think anyone will be able to answer your question without attempting to reproduce it. And also even checking and posting here "yep it looks fixed in main" is one of the best contributions you can make.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CLI command only] Importing `Variable` from `airflow.sdk` causes an `ImportError: cannot import name 'SUPERVISOR_COMMS'` for top-level Variable when running `airflow dags reserialize` [airflow]

Posted by "opeida (via GitHub)" <gi...@apache.org>.
opeida commented on issue #48554:
URL: https://github.com/apache/airflow/issues/48554#issuecomment-2919179641

   I faced the same ImportError trying to evaluate `LazyXComSequence(<unevaluated length>)` in the reduce function during dynamic task mapping. Running with `dag.test()`, LocalExecutor, Airflow 3.0.1.
   ```Python
   # Reduce function
   @task
   def send_notifications(data_to_notificate: list[tuple[Rule, ObjectStoragePath]], **context) -> None:
       for rule, input_file in data_to_notificate: # The error raises here
           ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org