You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "oggers (via GitHub)" <gi...@apache.org> on 2025/10/03 07:30:34 UTC
[I] airflow-db-not-allowed error when importing PostgresqlHook in a jupyter notebook and executing with PapermillOperator [airflow]
oggers opened a new issue, #56356:
URL: https://github.com/apache/airflow/issues/56356
### Apache Airflow Provider(s)
papermill
### Versions of Apache Airflow Providers
apache-airflow-providers-papermill==3.11.3
### Apache Airflow version
3.0.6
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### What happened
I have a jupyter notebook that contains:
```
from airflow.providers.postgres.hooks.postgres import PostgresHook
```
when executing this notebook with a PapermillOperator I get the following error:
```
---------------------------------------------------------------------------
ArgumentError Traceback (most recent call last)
Cell In[1], line 1
----> 1 from airflow.providers.postgres.hooks.postgres import PostgresHook
File [~/.local/lib/python3.12/site-packages/airflow/__init__.py:79](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/airflow/__init__.py#line=78)
73 # Perform side-effects unless someone has explicitly opted out before import
74 # WARNING: DO NOT USE THIS UNLESS YOU REALLY KNOW WHAT YOU'RE DOING.
75 # This environment variable prevents proper initialization, and things like
76 # configs, logging, the ORM, etc. will be broken. It is only useful if you only
77 # access certain trivial constants and free functions (e.g. `__version__`).
78 if not os.environ.get("_AIRFLOW__AS_LIBRARY", None):
---> 79 settings.initialize()
81 # Things to lazy import in form {local_name: ('target_module', 'target_name', 'deprecated')}
82 __lazy_imports: dict[str, tuple[str, str, bool]] = {
83 "DAG": (".models.dag", "DAG", False),
84 "Asset": (".assets", "Asset", False),
(...) 89 "Dataset": (".sdk.definitions.asset", "Dataset", True),
90 }
File [~/.local/lib/python3.12/site-packages/airflow/settings.py:618](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/airflow/settings.py#line=617), in initialize()
616 is_worker = os.environ.get("_AIRFLOW__REEXECUTED_PROCESS") == "1"
617 if not is_worker:
--> 618 configure_orm()
619 configure_action_logging()
621 # mask the sensitive_config_values
File [~/.local/lib/python3.12/site-packages/airflow/settings.py:359](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/airflow/settings.py#line=358), in configure_orm(disable_connection_pool, pool_class)
353 if SQL_ALCHEMY_CONN.startswith("sqlite"):
354 # FastAPI runs sync endpoints in a separate thread. SQLite does not allow
355 # to use objects created in another threads by default. Allowing that in test
356 # to so the `test` thread and the tested endpoints can use common objects.
357 connect_args["check_same_thread"] = False
--> 359 engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args, future=True)
360 async_engine = create_async_engine(SQL_ALCHEMY_CONN_ASYNC, future=True)
361 AsyncSession = sessionmaker(
362 bind=async_engine,
363 autocommit=False,
(...) 366 expire_on_commit=False,
367 )
File <string>:2, in create_engine(url, **kwargs)
File [~/.local/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py:375](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py#line=374), in deprecated_params.<locals>.decorate.<locals>.warned(fn, *args, **kwargs)
368 if m in kwargs:
369 _warn_with_version(
370 messages[m],
371 versions[m],
372 version_warnings[m],
373 stacklevel=3,
374 )
--> 375 return fn(*args, **kwargs)
File [~/.local/lib/python3.12/site-packages/sqlalchemy/engine/create.py:514](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/sqlalchemy/engine/create.py#line=513), in create_engine(url, **kwargs)
511 kwargs.pop("empty_in_strategy", None)
513 # create url.URL object
--> 514 u = _url.make_url(url)
516 u, plugins, kwargs = u._instantiate_plugins(kwargs)
518 entrypoint = u._get_entrypoint()
File [~/.local/lib/python3.12/site-packages/sqlalchemy/engine/url.py:738](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/sqlalchemy/engine/url.py#line=737), in make_url(name_or_url)
725 """Given a string or unicode instance, produce a new URL instance.
726
727
(...) 734
735 """
737 if isinstance(name_or_url, util.string_types):
--> 738 return _parse_url(name_or_url)
739 else:
740 return name_or_url
File [~/.local/lib/python3.12/site-packages/sqlalchemy/engine/url.py:799](http://jupyter-new.local.zabik/user/airflow/lab/tree/bizak/dags/items_available/~/.local/lib/python3.12/site-packages/sqlalchemy/engine/url.py#line=798), in _parse_url(name)
796 return URL.create(name, **components)
798 else:
--> 799 raise exc.ArgumentError(
800 "Could not parse SQLAlchemy URL from string '%s'" % name
801 )
ArgumentError: Could not parse SQLAlchemy URL from string 'airflow-db-not-allowed:///'
```
### What you think should happen instead
It should allow using dbhooks and airflow connection inside a jupyter notebook.
### How to reproduce
Create a notebook with only:
```
from airflow.providers.postgres.hooks.postgres import PostgresHook
```
and execute it with **PapermillOperator**
Other hooks like **MsSqlHook** also fails.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow-db-not-allowed error when importing PostgresqlHook in a jupyter notebook and executing with PapermillOperator [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #56356: airflow-db-not-allowed error when importing PostgresqlHook in a jupyter notebook and executing with PapermillOperator
URL: https://github.com/apache/airflow/issues/56356
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow-db-not-allowed error when importing PostgresqlHook in a jupyter notebook and executing with PapermillOperator [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #56356:
URL: https://github.com/apache/airflow/issues/56356#issuecomment-3398487850
This is expected. Airflow worker does not have access to Airflow DB any more. You seem to want to use Airfow DB by the hook of yours - if you want to do it, you need to define the connection. Apparently you want to initialize airflow from within the jupyter notebook and for some reason the notebook strips `_AIRFLOW__REEXECUTED_PROCESS` that should be set inside your worker - make sure it is not stripped - otherwise it will try to reinitialize airflow ORM which should not happen in worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow-db-not-allowed error when importing PostgresqlHook in a jupyter notebook and executing with PapermillOperator [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #56356:
URL: https://github.com/apache/airflow/issues/56356#issuecomment-3364600386
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org