You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/09 13:00:36 UTC
[GitHub] [airflow] potiuk opened a new pull request, #28259: Speed up most Users/Role CLI commands
potiuk opened a new pull request, #28259:
URL: https://github.com/apache/airflow/pull/28259
Originally those command were initializing whole Airflow Flask webserver, which was much more than what was needed - we only need to initialize Flask Appblication Builder. In case you have slow DB this limits significantly not only a number of imported classses but also a number of DB connections.
In some cases the speed of those CLI commands will visibly go down from 10s of seonds to individual seconds (3x - 5x times).
Follow up after #28242, #28244
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28259: Speed up most Users/Role CLI commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28259:
URL: https://github.com/apache/airflow/pull/28259#issuecomment-1344298757
I actually improved that quite a bit - by adding a separately imported util package with generator and moving all the Flask/AirflowAppBuilder imports local imposrts to be done as late as possible - in this case we only incure the penalty of Flask import and related classes when we are running the commands that need them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28259: Speed up most Users/Role CLI commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #28259:
URL: https://github.com/apache/airflow/pull/28259#discussion_r1044495547
##########
airflow/cli/commands/user_command.py:
##########
@@ -49,42 +48,47 @@ class UserSchema(Schema):
@suppress_logs_and_warning
def users_list(args):
"""Lists users at the command line."""
- appbuilder = cached_app().appbuilder
- users = appbuilder.sm.get_all_users()
- fields = ["id", "username", "email", "first_name", "last_name", "roles"]
+ from airflow.utils.cli_app_builder import get_application_builder
- AirflowConsole().print_as(
- data=users, output=args.output, mapper=lambda x: {f: x.__getattribute__(f) for f in fields}
- )
+ with get_application_builder() as appbuilder:
+ users = appbuilder.sm.get_all_users()
+ fields = ["id", "username", "email", "first_name", "last_name", "roles"]
+
+ AirflowConsole().print_as(
+ data=users, output=args.output, mapper=lambda x: {f: x.__getattribute__(f) for f in fields}
+ )
@cli_utils.action_cli(check_db=True)
def users_create(args):
"""Creates new user in the DB."""
- appbuilder = cached_app().appbuilder
- role = appbuilder.sm.find_role(args.role)
- if not role:
- valid_roles = appbuilder.sm.get_all_roles()
- raise SystemExit(f"{args.role} is not a valid role. Valid roles are: {valid_roles}")
-
- if args.use_random_password:
- password = "".join(random.choice(string.printable) for _ in range(16))
- elif args.password:
- password = args.password
- else:
- password = getpass.getpass("Password:")
- password_confirmation = getpass.getpass("Repeat for confirmation:")
- if password != password_confirmation:
- raise SystemExit("Passwords did not match")
-
- if appbuilder.sm.find_user(args.username):
- print(f"{args.username} already exist in the db")
- return
- user = appbuilder.sm.add_user(args.username, args.firstname, args.lastname, args.email, role, password)
- if user:
- print(f'User "{args.username}" created with role "{args.role}"')
- else:
- raise SystemExit("Failed to create user")
+ from airflow.utils.cli_app_builder import get_application_builder
+
+ with get_application_builder() as appbuilder:
Review Comment:
I do not want to make any more changes. Pareto's rule again - 20% of cost getting 80% gain.
Sooner or later we WILL get rid of Fab in t's current form so those commands should be refactored then. For now - I think we have to live what we have and make our dependencies smaller and smaller to the point that such a replacement will be easy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28259: Speed up most Users/Role CLI commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28259:
URL: https://github.com/apache/airflow/pull/28259#issuecomment-1344434642
Some tests will need fixes - of course :). Will do it tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28259: Speed up most Users/Role CLI commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28259:
URL: https://github.com/apache/airflow/pull/28259#issuecomment-1344282186
Hey @malthe - I'd appreciate if you can double check those commands on your system and see what kind of improvements you can get.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] malthe commented on a diff in pull request #28259: Speed up most Users/Role CLI commands
Posted by GitBox <gi...@apache.org>.
malthe commented on code in PR #28259:
URL: https://github.com/apache/airflow/pull/28259#discussion_r1044469613
##########
airflow/cli/commands/user_command.py:
##########
@@ -49,42 +48,47 @@ class UserSchema(Schema):
@suppress_logs_and_warning
def users_list(args):
"""Lists users at the command line."""
- appbuilder = cached_app().appbuilder
- users = appbuilder.sm.get_all_users()
- fields = ["id", "username", "email", "first_name", "last_name", "roles"]
+ from airflow.utils.cli_app_builder import get_application_builder
- AirflowConsole().print_as(
- data=users, output=args.output, mapper=lambda x: {f: x.__getattribute__(f) for f in fields}
- )
+ with get_application_builder() as appbuilder:
+ users = appbuilder.sm.get_all_users()
+ fields = ["id", "username", "email", "first_name", "last_name", "roles"]
+
+ AirflowConsole().print_as(
+ data=users, output=args.output, mapper=lambda x: {f: x.__getattribute__(f) for f in fields}
+ )
@cli_utils.action_cli(check_db=True)
def users_create(args):
"""Creates new user in the DB."""
- appbuilder = cached_app().appbuilder
- role = appbuilder.sm.find_role(args.role)
- if not role:
- valid_roles = appbuilder.sm.get_all_roles()
- raise SystemExit(f"{args.role} is not a valid role. Valid roles are: {valid_roles}")
-
- if args.use_random_password:
- password = "".join(random.choice(string.printable) for _ in range(16))
- elif args.password:
- password = args.password
- else:
- password = getpass.getpass("Password:")
- password_confirmation = getpass.getpass("Repeat for confirmation:")
- if password != password_confirmation:
- raise SystemExit("Passwords did not match")
-
- if appbuilder.sm.find_user(args.username):
- print(f"{args.username} already exist in the db")
- return
- user = appbuilder.sm.add_user(args.username, args.firstname, args.lastname, args.email, role, password)
- if user:
- print(f'User "{args.username}" created with role "{args.role}"')
- else:
- raise SystemExit("Failed to create user")
+ from airflow.utils.cli_app_builder import get_application_builder
+
+ with get_application_builder() as appbuilder:
Review Comment:
As a general code comment, splitting this sort of logic out into a separate function would alleviate having these big diffs where really it's a context manager that's changing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28259: Speed up most Users/Role CLI commands
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28259:
URL: https://github.com/apache/airflow/pull/28259#issuecomment-1345234968
One more "cached_app()" to remove :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #28259: Speed up most Users/Role CLI commands
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #28259:
URL: https://github.com/apache/airflow/pull/28259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org