You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "michael-s-molina (via GitHub)" <gi...@apache.org> on 2024/08/08 14:53:13 UTC
[PR] chore: Logs the duration of migrations execution [superset]
michael-s-molina opened a new pull request, #29893:
URL: https://github.com/apache/superset/pull/29893
### SUMMARY
Depending on the amount of data an organization has, the migration scripts can take a long time to run. This PR adds the duration of the complete migration process to help administrators to estimate downtime for upgrades/downgrades. Duration is only logged in case all migrations successfully complete.
### TESTING INSTRUCTIONS
CI should be sufficient.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Logs the duration of migrations execution [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #29893:
URL: https://github.com/apache/superset/pull/29893#discussion_r1710854888
##########
superset/migrations/env.py:
##########
@@ -54,6 +54,12 @@
# ... etc.
+def print_duration(start_time: float) -> None:
+ duration = time.time() - start_time
+ formatted_time = time.strftime("%H:%M:%S", time.gmtime(duration))
+ logger.info(f"Migration scripts completed. Duration: {formatted_time}")
Review Comment:
Nitpick but usually you should let the logger format the message for you rather than doing it yourself, since the logger will do it lazily and avoid having to do the formatting if INFO level logs are off, for example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Logs the duration of migrations execution [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina merged PR #29893:
URL: https://github.com/apache/superset/pull/29893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Logs the duration of migrations execution [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #29893:
URL: https://github.com/apache/superset/pull/29893#issuecomment-2276045509
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/29893?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `11 lines` in your changes missing coverage. Please review.
> Project coverage is 65.05%. Comparing base [(`76d897e`)](https://app.codecov.io/gh/apache/superset/commit/76d897eaa2f9e137102bc194c2e3109c29d0348f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`014a294`)](https://app.codecov.io/gh/apache/superset/commit/014a2946736821b7108f23df07b2e029cf25d520?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 573 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/superset/pull/29893?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/migrations/env.py](https://app.codecov.io/gh/apache/superset/pull/29893?src=pr&el=tree&filepath=superset%2Fmigrations%2Fenv.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvbWlncmF0aW9ucy9lbnYucHk=) | 0.00% | [11 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/29893?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #29893 +/- ##
==========================================
+ Coverage 60.48% 65.05% +4.57%
==========================================
Files 1931 527 -1404
Lines 76236 38042 -38194
Branches 8568 0 -8568
==========================================
- Hits 46114 24750 -21364
+ Misses 28017 13292 -14725
+ Partials 2105 0 -2105
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/29893/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/29893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.01% <0.00%> (-0.15%)` | :arrow_down: |
| [javascript](https://app.codecov.io/gh/apache/superset/pull/29893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/29893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.56% <0.00%> (-0.25%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/29893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.05% <0.00%> (+1.57%)` | :arrow_up: |
| [unit](https://app.codecov.io/gh/apache/superset/pull/29893/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.28% <0.00%> (+2.66%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/29893?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Logs the duration of migrations execution [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #29893:
URL: https://github.com/apache/superset/pull/29893#discussion_r1711457409
##########
superset/migrations/env.py:
##########
@@ -54,6 +54,12 @@
# ... etc.
+def print_duration(start_time: float) -> None:
+ duration = time.time() - start_time
+ formatted_time = time.strftime("%H:%M:%S", time.gmtime(duration))
+ logger.info(f"Migration scripts completed. Duration: {formatted_time}")
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Logs the duration of migrations execution [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #29893:
URL: https://github.com/apache/superset/pull/29893#discussion_r1710854888
##########
superset/migrations/env.py:
##########
@@ -54,6 +54,12 @@
# ... etc.
+def print_duration(start_time: float) -> None:
+ duration = time.time() - start_time
+ formatted_time = time.strftime("%H:%M:%S", time.gmtime(duration))
+ logger.info(f"Migration scripts completed. Duration: {formatted_time}")
Review Comment:
Nitpick but usually you should let the logger format the message for you rather than doing it yourself, since the logger will do it lazily and avoid having to do the formatting if INFO leve llogs are off, for example.
##########
superset/migrations/env.py:
##########
@@ -117,6 +130,7 @@ def process_revision_directives( # pylint: disable=redefined-outer-name, unused
try:
with context.begin_transaction():
context.run_migrations()
+ print_duration(start_time)
Review Comment:
Should this be outside the transaction context? I assume the context manager here is going to commit the transaction on exit, so it's not really done until that happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Logs the duration of migrations execution [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #29893:
URL: https://github.com/apache/superset/pull/29893#issuecomment-2278604268
> [optional] a context decorator could be nice and simple for this:
That would be nice. We can think about this when creating the helper functions for the migrations as part of [[SIP-142] Improving Database Migration Management](https://github.com/apache/superset/issues/29546)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Logs the duration of migrations execution [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #29893:
URL: https://github.com/apache/superset/pull/29893#discussion_r1711432757
##########
superset/migrations/env.py:
##########
@@ -117,6 +130,7 @@ def process_revision_directives( # pylint: disable=redefined-outer-name, unused
try:
with context.begin_transaction():
context.run_migrations()
+ print_duration(start_time)
Review Comment:
Good point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org