You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/08/11 07:02:30 UTC

[GitHub] [airflow] potiuk opened a new pull request, #33309: Reuse _run_task_session in mapped render_template_fields

potiuk opened a new pull request, #33309:
URL: https://github.com/apache/airflow/pull/33309

   The `render_template_fields` method of mapped operator needs to use database session object to render mapped fields, but it cannot get the session passed by @provide_session decorator, because it is used in derived classes and we cannot change the signature without impacting those classes.
   
   So far it was done by creating new session in mapped_operator, but it has the drawback of creating an extra session while one is already created (remnder_template_fields is always run in the context of task run and it always has a session created already in _run_raw_task). It also causes problems in our tests where two opened database session accessed database at the same time and it cases sqlite exception on concurrent access and mysql error on running operations out of sync - likely when the same object was modified in both sessions.
   
   This PR changes the approach - rather than creating a new session in the mapped_operator, we are retrieving the session from one stored by the _run_raw_task. It is done by context manager and adequate protection has been added to make sure that:
   
   a) the call is made within the context manager
   b) context manageer is never initialized twice in the same
      call stack
   
   After this change, resources used by running task will be smaller, and mapped tasks will not always open 2 DB sesions.
   
   Fixes: #33178
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1700695632

   The only thing I am wondering is whether the getter should return a new session if one is not set previously. Currently it would raise an error in this case, which is arguably a compatibility issue; we can probably categorise this change as an enhancement and supply a significant change notice in 2.8.0, but for 2.7.x I would prefer this to not break unexpected code paths in user code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1675157093

   All passed this time. Trying again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1678487753

   The main reason was the signature of the render_templelate method that could not be changed for the regular non-mapped operator. 
   
   At least that"s what the comment says in the original comment.
   
   The regular operators do not need the session at all. And we are extending the regular operator methods. So this is the main reason why we cannot pass the session directly. And the only one.
   
   The way it was implemented created (and not closed) a new session. It did not close it because it was  hard with the recursive patterns of template rendering.
   
   The way I implemented it now - we so not create new session. But use the one that comes with the task - and close it when run method completes. This Is the same as creating session but without the overhead of creating a new one - just reusing one that we already have 
   
   So basically we are doing what we've been doing already but with one session rather than two - and we are closing the session when. It is not needed. This is the saving we have.
   
   Do you see any flaw with this reasoning? Do you think it's not worthy a saving of extra session and db connection created always when mapped operator is running? I think it is quite significant improvement TBH, without -seemingly - any drawback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1678489645

   Or maybe you think we can change the signature ? I believe (on the phone now) the original comment mentioned that the signature cannot be changed because the methods that would have to be changed are already in the wild being extended by existing operators and by changing the signature and adding session would breake those operators. 
   
   I would very much prefer it this way as well if we can do it - do you think it is possible by looking at the comment ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields
URL: https://github.com/apache/airflow/pull/33309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields
URL: https://github.com/apache/airflow/pull/33309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1678419127

   I’m having second thoughts. If this works, why can’t we just pass the `session` all the way from `_run_raw_task` into `render_template_fields` instead? The fact we could not do this previously seems to hint that using a context manager to clean this up will not work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1686013452

   @uranusjr ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1675223926

   Ok. We have one here, but it's unrelated to mapped operator (that's the first time I see it) https://github.com/apache/airflow/issues/33323  - another one to investigate.
   
   This is the third complete test without re-occurence of the "mapped" errors - usually we would have 3-4 of those already, I consider the "mapped" as fixed but I will try few more times.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33309: Save connection resources in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33309:
URL: https://github.com/apache/airflow/pull/33309#discussion_r1293274644


##########
airflow/cli/commands/task_command.py:
##########
@@ -634,6 +635,12 @@ def task_test(args, dag: DAG | None = None) -> None:
                 session.delete(ti.dag_run)
 
 
+@provide_session
+def render_template_with_session(ti: TaskInstance, session=NEW_SESSION):
+    with set_current_task_instance_session(session=session):
+        ti.render_templates()

Review Comment:
   I can also make it a nested function inside "task_render", but I am rather sceptical about adding functions inside functions. We could also potentially turn "provide_session" into a generic context manager but that would likely need some modifications to it and it would be introducing a new way of managing and allocating sessions - I would rather not introduce a new way of managing sessions but I would rely on how we do it elsewhere. 
   
   I considered all those options and extracting the method the way I did, sounded like the most "straightforward" solution.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1674812225

   Hey Herel. While doing my "flaky test squashing" quest, and investigating the flaky "session" test #33178 I deduced that there must be something deep in the mapped operator implementation that causes sessions leak during the tests, that might actually be also causing similar problems in production.
   
   It turned out that the problems are (almost for sure) caused by the way we retrieved (and not closed/discarded) DB session while rendering templates in mapped tasks. I discussed it with @uranusjr and while doing some experiments (mainly in "Chasing leaking sessions" PR https://github.com/apache/airflow/pull/33259 I realized that it has an unintended side effect that we are opening more ssessions that necessary when runnning mapped task - basically we always open one session for the "_run_raw_task" and then another session just to "render templates". 
   
   There is a good reason for doing it (we cannot easily pass sesssion through usual @provide_session mechanism because "render_template" method's signature need to be compatible with non-mapped object, but the way we've done that was pretty reasource-heavy and brittle. It opened a new session and basically it had not closed the session, because of recursive behaviour it was quite possible the session could be close too early. 
   
   In the discussion with @uranusjr https://github.com/apache/airflow/issues/33178#issuecomment-1674206630  it turned out that this method is only actually called in the context of `_run_raw_task` - so I figured that we can do it much more efficiently and without the "brittleness" - i.e. we could have a context manager that will store the "task" session in the global data and then render_task_template of mapped operator could use that globally stored session rather than create a new one.
   
   I think it's much safer/stable and less resource hungry (just single session needed for the mapped task, not two) - but also far more robust (the context manager will close the session when it leaves the _run_raw_task. It's also is going to solve the test flakiness I think (i will close /reopen this PR few times to see if the problem is gone).
   
   I cannot see any bad side-effects of the change - I already fixed all the tests that need the context manager and added the context manager to the CLI method that also uses render_templates (the only place besides "_run_raw_task" that apparently uses rendering). 
   
   I'd love those who have a bit more experience in those parts (@ashb @uranusjr) to take a look and see if I have not missed anything - this one might be a good idea for 2.7.1 (or maybe even we could add it to rc2 of 2.7.0 if we have one).
   
   Looking forward to comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields
URL: https://github.com/apache/airflow/pull/33309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #33309:
URL: https://github.com/apache/airflow/pull/33309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1682658572

   Yes - it looks like it can't be easily changed. the "render_template_fields" is part of the public interface of the BaseOperator, and chaning it by adding a session - would not really work as even in our community operators the method has been already overridden (the method is caled by basenotifier for example and in sql_to_slack operator - but generally it looks like we cannot **just** pass a session there : here is the original comment.
   
   ```
     def render_template_fields(
           self,
           context: Context,
           jinja_env: jinja2.Environment | None = None,
       ) -> None:
           # Ideally we'd like to pass in session as an argument to this function,
           # but we can't easily change this function signature since operators
           # could override this. We can't use @provide_session since it closes and
           # expunges everything, which we don't want to do when we are so "deep"
           # in the weeds here. We don't close this session for the same reason.
   ```
   
   Do you have some other suggestions/ideas how we can do it withouth some kind of local state where we could keep it to pass it down @uranusjr ?  I am out of ideas (and I think that the session/resource leak is quite problematic and we should fix it somehow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1676425530

   cc: @uranusjr @ashb - would really love to get it in - this one seems to stabilize a lot, still there are some cases where tests are flaky but they are few and far between comparing to recently observed flakiness (+ it also saves resources for mapped tasks).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields
URL: https://github.com/apache/airflow/pull/33309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33309:
URL: https://github.com/apache/airflow/pull/33309#discussion_r1293729443


##########
airflow/cli/commands/task_command.py:
##########
@@ -634,6 +635,12 @@ def task_test(args, dag: DAG | None = None) -> None:
                 session.delete(ti.dag_run)
 
 
+@provide_session
+def render_template_with_session(ti: TaskInstance, session=NEW_SESSION):
+    with set_current_task_instance_session(session=session):
+        ti.render_templates()

Review Comment:
   Right.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1706741295

   Thanks @uranusjr !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1675094686

   So far so good. No trace of session problems 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33309: Save connection resources in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33309:
URL: https://github.com/apache/airflow/pull/33309#discussion_r1293270209


##########
airflow/cli/commands/task_command.py:
##########
@@ -634,6 +635,12 @@ def task_test(args, dag: DAG | None = None) -> None:
                 session.delete(ti.dag_run)
 
 
+@provide_session
+def render_template_with_session(ti: TaskInstance, session=NEW_SESSION):
+    with set_current_task_instance_session(session=session):
+        ti.render_templates()

Review Comment:
   To use decorator applired to a minimum scope it is needed. We do not need to keep session open after we already rendered the templates - i.e when we print the rendeder template values.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #33309:
URL: https://github.com/apache/airflow/pull/33309#discussion_r1292972076


##########
airflow/cli/commands/task_command.py:
##########
@@ -634,6 +635,12 @@ def task_test(args, dag: DAG | None = None) -> None:
                 session.delete(ti.dag_run)
 
 
+@provide_session
+def render_template_with_session(ti: TaskInstance, session=NEW_SESSION):
+    with set_current_task_instance_session(session=session):
+        ti.render_templates()

Review Comment:
   Why does this need a separate function?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1701335198

   > The only thing I am wondering is whether the getter should return a new session if one is not set previously. Currently it would raise an error in this case, which is arguably a compatibility issue; we can probably categorise this change as an enhancement and supply a significant change notice in 2.8.0, but for 2.7.x I would prefer this to not break unexpected code paths in user code.
   
   If that's the only worry @uranusjr - I fixed it. Instead of raising errror, I print a warning with detailed stack trace and asking the user to report it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields
URL: https://github.com/apache/airflow/pull/33309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields
URL: https://github.com/apache/airflow/pull/33309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1676118052

   Reopened to rebuild


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1678486007

   The main reason was the signature of the render_templelate method that could not be changed for the regular non-mapped operator. 
   
   At least that"s what the comment says in the original comment.
   
   The regular operators do not need the session at all. And we are extending the regular operator methods. So this is the main reason why we cannot pass the session directly. And the only one.
   
   The way it was implemented created (and not closed) a new session. It did not close it because it was  hard with the recursive patterns of template rendering.
   
   The way I implemented it now - we so not create new session. But use the one that comes with the task - and close it when run method completes. This Is the same as creating session but without the overhead of creating a new one - just reusing one that we already have 
   
   So basically we are doing what we've been doing already but with one session rather than two - and we are closing the session when. It is not needed. This is the saving we have.
   
   Do you see any flaw with this reasoning? Do you think it's not worthy a saving of extra session and db connection created always when mapped operator is running? I think it is quite significant improvement TBH, without -seemingly - any drawback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields
URL: https://github.com/apache/airflow/pull/33309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on a diff in pull request #33309: Limit connection resources used in mapped tasks

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #33309:
URL: https://github.com/apache/airflow/pull/33309#discussion_r1293681019


##########
airflow/cli/commands/task_command.py:
##########
@@ -634,6 +635,12 @@ def task_test(args, dag: DAG | None = None) -> None:
                 session.delete(ti.dag_run)
 
 
+@provide_session
+def render_template_with_session(ti: TaskInstance, session=NEW_SESSION):
+    with set_current_task_instance_session(session=session):
+        ti.render_templates()

Review Comment:
   There’s `create_session`, a decorator is not needed.
   
   ```python
   with create_session() as session, set_current_task_instance_session(session=session):
       ti.render_templates()
   ```



##########
airflow/cli/commands/task_command.py:
##########
@@ -634,6 +635,12 @@ def task_test(args, dag: DAG | None = None) -> None:
                 session.delete(ti.dag_run)
 
 
+@provide_session
+def render_template_with_session(ti: TaskInstance, session=NEW_SESSION):
+    with set_current_task_instance_session(session=session):
+        ti.render_templates()

Review Comment:
   There’s `create_session`, a decorator is not needed.
   
   ```python
   with create_session() as session, set_current_task_instance_session(session=session):
       ti.render_templates()
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33309:
URL: https://github.com/apache/airflow/pull/33309#discussion_r1293735818


##########
airflow/cli/commands/task_command.py:
##########
@@ -634,6 +635,12 @@ def task_test(args, dag: DAG | None = None) -> None:
                 session.delete(ti.dag_run)
 
 
+@provide_session
+def render_template_with_session(ti: TaskInstance, session=NEW_SESSION):
+    with set_current_task_instance_session(session=session):
+        ti.render_templates()

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1676329891

   Ok. We had one more failure with sqlite last time, but it was in teardown, so a bit different, and I **think** it was caused by lack of passing session to create_dagrun. I added session to it, and will continue rebasing it. 
   
   One of the followups of this one might be a global attempt to make sure in all our tests whenever ti.run() or dag_maker.create_dagrun() are called we might want to  add some pre-commit/ast to force it's usage


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1703304398

   @uranusjr ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Limit connection resources used in mapped tasks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1695118749

   Hey @uranusjr -> I would ***REALLY*** love that one to get into 2.7.1 - and basically I run it so many times (and I think the proposal is pretty sound) that I have a reason to believe it does solve some of the resourcing issues. We have a number of reports indicating that Airflow became much more resource hungry in recent versions. I am not sure if that could be one of the reason  but I am quite certain it is going to decrease the amount of resources needed by the workers. 
   
   It just basically works for your approval - maybe we can ask others to review it, but I think it's not a good idea to keep it in the limbo forever.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1676118013

   All green this time. I will keep on re-running but seems that the basic problem is solved. @uranusjr - if you agree, we seem to have an RC2 for 2.7.0 is coming, do you think it's good to get it for RC2?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1676399124

   Seems that there is yet another reason for the sessions. The quest is not over yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33309:
URL: https://github.com/apache/airflow/pull/33309#issuecomment-1674813308

   Reopened to see if flakiness reappears.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #33309: Reuse _run_task_session in mapped render_template_fields
URL: https://github.com/apache/airflow/pull/33309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org