You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2025/10/29 10:33:14 UTC
[PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
amoghrajesh opened a new pull request, #57479:
URL: https://github.com/apache/airflow/pull/57479
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
closes: https://github.com/apache/airflow/issues/57157
## Aim
Airflow 2 has many users, and during the Airflow 3 upgrade process, one of the most common questions raised (including at Airflow Summit) is: "How do I access the metadata database from my tasks now that direct database access is blocked?"
This PR addresses this frequently asked question by expanding Step 5 of the upgrading guide with a few options.
The documentation now aims to provide a clear, actionable guidance that will help lot of users migrating from Airflow 2.x to 3.x avoid confusion and migration blockers.
## Testing details for each method
I have added in 3 methods which I have tested personally.
### 1. Airflow Python Client Approach
I wrote a simple task like this based on the documentation, and I was able to achieve this:
```python
def fetch_all_connections(
host: str | None = None,
username: str | None = None,
password: str | None = None,
limit: int = 100,
) -> list[dict]:
host = host or os.environ.get("AIRFLOW_API_HOST", "http://localhost:28080")
username = username or os.environ.get("AIRFLOW_USERNAME", "admin")
password = password or os.environ.get("AIRFLOW_PASSWORD", "admin")
# Get access token
access_token = get_airflow_access_token(host, username, password)
print("Access token", access_token)
# Configure client
configuration = airflow_client.client.Configuration(
host=host,
access_token=access_token,
)
# Fetch connections
with airflow_client.client.ApiClient(configuration) as api_client:
conn_api = connection_api.ConnectionApi(api_client)
response = conn_api.get_connections(limit=limit)
print(response)
# Convert to list of dicts
connections = []
for conn in response.connections:
connections.append(connections)
return connections
```
Result:
```shell script
connections=[ConnectionResponse(conn_type='http', connection_id='my_conn', description=None, extra=None, host='myhost', login='login1', password='***', port=1025, var_schema='default')] total_entries=1
```
### 2. PostgresHook/DbApiHook Approach
Tried with PostgresHook and I was able to achieve this:
Task:
```python
@task
def query_connections():
hook = PostgresHook(postgres_conn_id="metadata_postgres")
records = hook.get_records(
sql="""
SELECT conn_id
FROM connection
"""
)
return records
```
<img width="1724" height="883" alt="image" src="https://github.com/user-attachments/assets/2f30a10f-6fb8-4269-8b8d-1f20037a118f" />
### 3. Direct Database Access (Last Resort)
Created a temporary connection and called it `metadata_db_connect_direct`:
```shell script
export METADATA_DB_CONN_DIRECT=postgresql+psycopg2://postgres:airflow@postgres/airflo
```
Wrote a helper that can create a session object for us:
```shell script
def get_direct_db_session():
conn_string = os.environ.get("METADATA_DB_CONN_DIRECT")
if not conn_string:
raise RuntimeError(
"METADATA_DB_CONN_DIRECT environment variable not set. "
"Cannot create direct database session."
)
engine = create_engine(
conn_string,
)
SessionLocal = sessionmaker(bind=engine)
session = SessionLocal()
return session
```
Task:
```python
@task
def my_task():
session = get_direct_db_session()
from airflow.models import Connection
conns = session.query(Connection).all()
print("Queries and found these conns", conns)
```
And I was able to get this:
<img width="1724" height="883" alt="image" src="https://github.com/user-attachments/assets/1d61616d-7d40-4653-92a6-0c8972e2e671" />
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2472758188
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,76 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure there are no direct db access.
Review Comment:
Handled
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,76 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
Review Comment:
Handled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2473320444
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,83 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3, operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure that there are no direct database access calls.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+
+If you have custom operators or task code that previously accessed the metadata database directly, you need to migrate to one of the following recommended approaches:
Review Comment:
```suggestion
If you have custom operators or task code that previously accessed the metadata database directly, you must migrate to one of the following recommended approaches:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2476650762
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,83 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3, operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure that there are no direct database access calls.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+
+If you have custom operators or task code that previously accessed the metadata database directly, you need to migrate to one of the following recommended approaches:
+
+Recommended Approach 1: Use Airflow Python Client
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Use the official `Airflow Python Client <https://github.com/apache/airflow-client-python>`_ to interact with
+Airflow metadata via REST API. The Python Client has APIs defined for most use cases, including DagRuns, TaskInstances, Variables, Connections, XComs, and more.
+
+Recommended Approach 2: Use DbApiHook (PostgresHook or MySqlHook)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Review Comment:
Same comment as above
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,83 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3, operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure that there are no direct database access calls.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+
+If you have custom operators or task code that previously accessed the metadata database directly, you need to migrate to one of the following recommended approaches:
+
+Recommended Approach 1: Use Airflow Python Client
Review Comment:
I plan to add a pro/con section to each approach today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #57479:
URL: https://github.com/apache/airflow/pull/57479#issuecomment-3476537647
One could really think seeing this recommendation "why the hell we've done all that efforts in the first place since now we recommend going back to what things were basically" (minus recommending a read-only access of course which is indeed a bit better).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #57479:
URL: https://github.com/apache/airflow/pull/57479#issuecomment-3476536584
And just to explain what I mean - I am not totally against it, this might be a viable solution - but I am totally against introducing advices that invalidate some of the promises we built Airflow 3 on - WITHOUT correcting the statements we already made, without correcting the comumunication we already made and without updating the documentaiton that contradicts that documentation and can trick our users into thinking that those promises we made are still kept even if they follow the recommendation.
I think - personally - we shoudl not make it as "recommended" solution - we should at the very least turn it into - "We've heard that some users are doing it and if you absolutely cannot do otherwise, this is one of the ways you can approach it- but be aware that promises we made here and there (with pointing axaclty where we made those promises) are not met and that you absolutely cannot expect any support whatsoever from us and that accessing the DB directly **will** break in the future versions of Airflow and that you should be prepared for it".
This should be a **VERY** clear message, and naming it as "recommendation" is quite an abomination if I have to be honest and basically laughing loud at all the efforts we made to do task isolation and related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2472504830
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,76 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure there are no direct db access.
Review Comment:
```suggestion
If you have custom operators, review the code to make sure that there are no direct database access calls.
```
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,76 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
Review Comment:
```suggestion
In Airflow 3, operators can not access the Airflow metadata database directly using database sessions.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #57479:
URL: https://github.com/apache/airflow/pull/57479#issuecomment-3484210883
> > That is the reason why I have not mentioned it was "recommended", but instead as "last resort" along with a warning:
>
> Well. The title of the chapter is (maybe we are not looking at the same version) or maybe you have not understand it:
>
> ```
> Recommended Approach 2: Use DbApiHook (PostgresHook or MySqlHook)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ```
>
> This is what I was referred to as something we should not recommend IMHO . The "last resort" is something that I would even not dream we can suggest (even as last resort) in the darkest nightmares :)
>
> I think the third option should be simply removed, and all the reservation about the "last resort" one moved to the 2nd method (and also have all the reservations about public interface). Then we might - I think - approach to something we **might** think as adding to our docs (but I would also love to make sure that this is passed through consensus on the devlist and we make everyone aware what we are signing up for and what are the consequences).
I was reading the third recommendation because you mentioned: `I am not sure that we should describe metadata DB access as "recommended" approach` which made me think its the third one. My bad.
> I think the third option should be simply removed, and all the reservation about the "last resort" one moved to the 2nd method (and also have all the reservations about public interface). Then we might - I think - approach to something we might think as adding to our docs (but I would also love to make sure that this is passed through consensus on the devlist and we make everyone aware what we are signing up for and what are the consequences)
Cool, let me first send a discussion over email and once I hear what people have to say, I can raise a lazy consensus.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2478627640
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,83 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3, operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure that there are no direct database access calls.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+
+If you have custom operators or task code that previously accessed the metadata database directly, you need to migrate to one of the following recommended approaches:
+
+Recommended Approach 1: Use Airflow Python Client
Review Comment:
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2473314455
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,83 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3, operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure that there are no direct database access calls.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
Review Comment:
```suggestion
In Airflow 3, operators cannot access the Airflow metadata database directly using database sessions.
If you have custom operators, review your code to ensure there are no direct database access calls.
You can follow examples in https://github.com/apache/airflow/issues/49187 to learn how to modify your code if needed.
```
ran through grammarly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2476201316
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,83 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3, operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure that there are no direct database access calls.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+
+If you have custom operators or task code that previously accessed the metadata database directly, you need to migrate to one of the following recommended approaches:
+
+Recommended Approach 1: Use Airflow Python Client
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Use the official `Airflow Python Client <https://github.com/apache/airflow-client-python>`_ to interact with
+Airflow metadata via REST API. The Python Client has APIs defined for most use cases, including DagRuns, TaskInstances, Variables, Connections, XComs, and more.
+
+Recommended Approach 2: Use DbApiHook (PostgresHook or MySqlHook)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Review Comment:
Same here
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,83 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3, operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure that there are no direct database access calls.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+
+If you have custom operators or task code that previously accessed the metadata database directly, you need to migrate to one of the following recommended approaches:
+
+Recommended Approach 1: Use Airflow Python Client
Review Comment:
Should we add a one-liner mentioning why this is a recommended approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #57479:
URL: https://github.com/apache/airflow/pull/57479#issuecomment-3481619025
> That is the reason why I have not mentioned it was "recommended", but instead as "last resort" along with a warning:
Well. The title of the chapter is (maybe we are not looking at the same version) or maybe you have not understand it:
```
Recommended Approach 2: Use DbApiHook (PostgresHook or MySqlHook)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```
This is what I was referred to as something we should not recommend IMHO . The "last resort" is somethign that I would even not dreamed we can suggest in the darkest nightmares :)
I think the third option should be simply removed, and all the reservation about the "last resort" one moved to the 2nd method (and also have all the reservations about public interface). Then we might - I think - approach to something we **might** think as adding to our docs (but I would also love to make sure that this is passed through consensus on the devlist and we make everyone aware what we are signing up for and what are the consequences).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #57479:
URL: https://github.com/apache/airflow/pull/57479#issuecomment-3540426079
New changes have been pushed after lazy consensus.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2476696279
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,83 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3, operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure that there are no direct database access calls.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+
+If you have custom operators or task code that previously accessed the metadata database directly, you need to migrate to one of the following recommended approaches:
+
+Recommended Approach 1: Use Airflow Python Client
Review Comment:
Pushed a change here: https://github.com/apache/airflow/pull/57479/commits/370940c1117ecd3fcacc2abe83cb5f36555a759c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #57479:
URL: https://github.com/apache/airflow/pull/57479#issuecomment-3560817417
Thanks Amogh! Looks great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2472750650
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,76 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure there are no direct db access.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+
+If you have custom operators or task code that previously accessed the metadata database directly, you need to migrate to one of the following recommended approaches:
+
+Recommended Approach 1: Use Airflow Python Client
Review Comment:
Good idea, handled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #57479:
URL: https://github.com/apache/airflow/pull/57479#discussion_r2472539854
##########
airflow-core/docs/installation/upgrading_to_airflow3.rst:
##########
@@ -195,9 +195,76 @@ Step 4: Install the Standard Provider
Step 5: Review custom operators for direct db access
----------------------------------------------------
-- In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
- If you have custom operators, review the code to make sure there are no direct db access.
- You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+In Airflow 3 operators can not access the Airflow metadata database directly using database sessions.
+If you have custom operators, review the code to make sure there are no direct db access.
+You can follow examples in https://github.com/apache/airflow/issues/49187 to find how to modify your code if needed.
+
+If you have custom operators or task code that previously accessed the metadata database directly, you need to migrate to one of the following recommended approaches:
+
+Recommended Approach 1: Use Airflow Python Client
Review Comment:
Add a link to Python client repo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #57479:
URL: https://github.com/apache/airflow/pull/57479#issuecomment-3479297326
Thanks for the careful review @potiuk!
I was hoping that something like this would come from someone atleast.
Few things re your comments:
> I am not sure that we should describe metadata DB access as "recommended" approach.
That is the reason why I have not mentioned it was "recommended", but instead as "last resort" along with a warning:
```shell script
**WARNING:** This approach bypasses Airflow's default protection and should only be used when the above methods cannot meet your requirements.
```
Regarding the statement about DB being a public interface, maybe I can clarify in this document that this recommendation has nothing to do with public interface of the database and can change at any time, but I believe I have not pointed out something as "suggestive" there, just a general -- use read only / rotated creds, export it and use it in task, that will remain same always?
> We should make it very clear that creating such connection and adding direct DB access is breaking task isolation principle we base all our security model on. Whenever me make such a recommendation here, we should immediately state that it is against our security model, or we shoudl update our model to explain that this is supported. Otherwise people will get confused and will think that what they see in the security model is "correct" - even if they follow the recommendation
I had added it in a warning, maybe I can clarify a bit more on that front? Just explicitly stating that this is against the security model of Airflow 3 and just a suggestion that you can use if nothing else works.
> we should explain the users what performance implications such 'recommended' way introduces. Such recommendation means coming back to Airflow 2 where we had each task opening a separate database connection and that dramatically changes performance characteristics of the system architecture of Airflow.
Every method has a pros/cons section and there is a con for this one:
- Bypasses Airflow's architectural protections and bad queries can affect performance of Airflow database
> I think - personally - we shoudl not make it as "recommended" solution - we should at the very least turn it into - "We've heard that some users are doing it and if you absolutely cannot do otherwise, this is one of the ways you can approach it- but be aware that promises we made here and there (with pointing axaclty where we made those promises) are not met and that you absolutely cannot expect any support whatsoever from us and that accessing the DB directly will break in the future versions of Airflow and that you should be prepared for it".
I have kind of pointed out that if nothing works, use this. Can you add some suggestions if you have some in mind re this?
> This should be a VERY clear message, and naming it as "recommendation" is quite an abomination if I have to be honest and basically laughing loud at all the efforts we made to do task isolation and related.
Just to be clear, I didn't explicitly call it a recommendation and added a "last resort" and "warning" too. I have no intentions of "trolling" any of the efforts involved in Airflow 3 as I was a big part of it too (my thoughts would still be the same even if I wasn't a part of task sdk etc).
I am totally OK to wipe off that recommendation if its that funny to have it in here, it is a solution that could work for people for whom nothing else works, so I vaguely defined it in here.
@potiuk if it makes sense, and if we (as a community) cannot resolve it over here, happy to start a devlist discussion for broader audience. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add migration options for Airflow 2 users using database access in tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #57479:
URL: https://github.com/apache/airflow/pull/57479
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org