You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/08 17:11:06 UTC

[GitHub] [airflow] dimberman opened a new pull request #8219: Add migration waiting script and log cleaner

dimberman opened a new pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219
 
 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#issuecomment-613851298
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=h1) Report
   > Merging [#8219](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b198a1fa94c44228dc7358552aeb6a5371ae0da2&el=desc) will **decrease** coverage by `0.62%`.
   > The diff coverage is `27.58%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8219/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #8219      +/-   ##
   ==========================================
   - Coverage   88.45%   87.83%   -0.63%     
   ==========================================
     Files         940      940              
     Lines       45354    45382      +28     
   ==========================================
   - Hits        40120    39860     -260     
   - Misses       5234     5522     +288     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `84.45% <19.23%> (-13.92%)` | :arrow_down: |
   | [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.29% <100.00%> (+0.01%)` | :arrow_up: |
   | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `95.45% <100.00%> (+0.21%)` | :arrow_up: |
   | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
   | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
   | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
   | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
   | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=footer). Last update [b198a1f...8a08498](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408674656
 
 

 ##########
 File path: scripts/include/clean-logs.sh
 ##########
 @@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+DIRECTORY=${AIRFLOW_HOME:-/usr/local/airflow}
+RETENTION=${AIRFLOW__LOG_RETENTION_DAYS:-15}
 
 Review comment:
   ```suggestion
   RETENTION="${AIRFLOW__LOG_RETENTION_DAYS:-15}"
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#issuecomment-613851298
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=h1) Report
   > Merging [#8219](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b198a1fa94c44228dc7358552aeb6a5371ae0da2&el=desc) will **decrease** coverage by `0.62%`.
   > The diff coverage is `27.58%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8219/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #8219      +/-   ##
   ==========================================
   - Coverage   88.45%   87.83%   -0.63%     
   ==========================================
     Files         940      940              
     Lines       45354    45382      +28     
   ==========================================
   - Hits        40120    39860     -260     
   - Misses       5234     5522     +288     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `84.45% <19.23%> (-13.92%)` | :arrow_down: |
   | [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.29% <100.00%> (+0.01%)` | :arrow_up: |
   | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `95.45% <100.00%> (+0.21%)` | :arrow_up: |
   | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
   | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
   | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
   | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
   | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=footer). Last update [b198a1f...8a08498](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
dimberman commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408835081
 
 

 ##########
 File path: airflow/cli/cli_parser.py
 ##########
 @@ -434,6 +434,12 @@ def __init__(self, flags=None, help=None, action=None, default=None, nargs=None,
 ARG_CFG_PATH = Arg(
     ("--cfg-path",),
     help="Path to config file to use instead of airflow.cfg")
+ARG_MIGRATION_TIMEOUT = Arg(
+    ("-t", "--migration-wait-timeout"),
+    help="timeout to wait for db to migrate ",
+    type=int,
+    default="0",
 
 Review comment:
   I think check-migrations is fine.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r405753019
 
 

 ##########
 File path: airflow/include/airflow_migration_spinner.py
 ##########
 @@ -0,0 +1,50 @@
+import argparse
+import importlib
+import logging
+import os
+import time
+
+from airflow import settings, version
+from alembic.config import Config
+from alembic.runtime.migration import MigrationContext
+from alembic.script import ScriptDirectory
+
+
+# package_dir is path of installed airflow in your virtualenv or system (site-packages)
+# we use it to find alembic.ini file
+def spinner(timeout):
+    package_dir = os.path.dirname(importlib.util.find_spec('airflow').origin)
+    directory = os.path.join(package_dir, 'migrations')
+    config = Config(os.path.join(package_dir, 'alembic.ini'))
+    config.set_main_option('script_location', directory)
+    config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN)
+    script_ = ScriptDirectory.from_config(config)
+
+    with settings.engine.connect() as connection:
+        context = MigrationContext.configure(connection)
+        ticker = 0
+        while True:
+            source_heads = set(script_.get_heads())
+            db_heads = set(context.get_current_heads())
+            if source_heads == db_heads:
+                logging.info('Airflow version: {}'.format(version.version))
+                logging.info('Current heads: {}'.format(db_heads))
+                break
+            elif ticker >= timeout:
+                raise TimeoutError("There are still unapplied migrations after {} "
+                                   "seconds".format(ticker, timeout))
+            ticker += 1
+            time.sleep(1)
+            logging.info('Waiting for migrations... {} second(s)'.format(ticker))
+
+
+def main():
 
 Review comment:
   This should be tied in to Airflow's existing command strucutre, so it is callable as `airflow db migration-status` or similar.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
dimberman commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408833820
 
 

 ##########
 File path: airflow/utils/db.py
 ##########
 @@ -561,6 +562,37 @@ def initdb():
     Base.metadata.create_all(settings.engine)  # pylint: disable=no-member
 
 
+def wait_for_migrations(timeout):
+    """
+    Function to wait for all airflow migrations to complete. Used for launching airflow in k8s
+    @param timeout:
+    @return:
+    """
+    from alembic.config import Config
+    from alembic.runtime.migration import MigrationContext
+
+    package_dir = os.path.dirname(importlib.util.find_spec('airflow').origin)
+    directory = os.path.join(package_dir, 'migrations')
+    config = Config(os.path.join(package_dir, 'alembic.ini'))
+    config.set_main_option('script_location', directory)
+    config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN)
+    script_ = ScriptDirectory.from_config(config)
 
 Review comment:
   Fixed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on issue #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#issuecomment-613813496
 
 
   @ashb changing that "and v" in the argument parsing broke a lot of things in non-trivial ways. I think if we want to fix how we parse commands that should be its own ticket.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
dimberman commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r407677032
 
 

 ##########
 File path: airflow/include/airflow_migration_spinner.py
 ##########
 @@ -0,0 +1,50 @@
+import argparse
+import importlib
+import logging
+import os
+import time
+
+from airflow import settings, version
+from alembic.config import Config
+from alembic.runtime.migration import MigrationContext
+from alembic.script import ScriptDirectory
+
+
+# package_dir is path of installed airflow in your virtualenv or system (site-packages)
+# we use it to find alembic.ini file
+def spinner(timeout):
+    package_dir = os.path.dirname(importlib.util.find_spec('airflow').origin)
+    directory = os.path.join(package_dir, 'migrations')
+    config = Config(os.path.join(package_dir, 'alembic.ini'))
+    config.set_main_option('script_location', directory)
+    config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN)
+    script_ = ScriptDirectory.from_config(config)
+
+    with settings.engine.connect() as connection:
+        context = MigrationContext.configure(connection)
+        ticker = 0
+        while True:
+            source_heads = set(script_.get_heads())
+            db_heads = set(context.get_current_heads())
+            if source_heads == db_heads:
+                logging.info('Airflow version: {}'.format(version.version))
+                logging.info('Current heads: {}'.format(db_heads))
+                break
+            elif ticker >= timeout:
+                raise TimeoutError("There are still unapplied migrations after {} "
+                                   "seconds".format(ticker, timeout))
+            ticker += 1
+            time.sleep(1)
+            logging.info('Waiting for migrations... {} second(s)'.format(ticker))
+
+
+def main():
 
 Review comment:
   Ok SG

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
dimberman commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408827758
 
 

 ##########
 File path: requirements/requirements-python3.6.txt
 ##########
 @@ -123,21 +123,21 @@ future-fstrings==1.2.0
 future==0.18.2
 gcsfs==0.6.1
 google-ads==4.0.0
-google-api-core==1.16.0
+google-api-core==1.17.0
 google-api-python-client==1.8.0
 google-auth-httplib2==0.0.3
 google-auth-oauthlib==0.4.1
-google-auth==1.13.1
+google-auth==1.14.0
 google-cloud-automl==0.10.0
 google-cloud-bigquery-datatransfer==1.0.0
 google-cloud-bigquery==1.24.0
 google-cloud-bigtable==1.2.1
-google-cloud-container==0.4.0
+google-cloud-container==0.5.0
 
 Review comment:
   When I ran the build the "generate requirements" step was failing. I'm honestly not sure why this was necessary either.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408673753
 
 

 ##########
 File path: airflow/utils/db.py
 ##########
 @@ -561,6 +562,37 @@ def initdb():
     Base.metadata.create_all(settings.engine)  # pylint: disable=no-member
 
 
+def wait_for_migrations(timeout):
+    """
+    Function to wait for all airflow migrations to complete. Used for launching airflow in k8s
+    @param timeout:
+    @return:
+    """
+    from alembic.config import Config
+    from alembic.runtime.migration import MigrationContext
+
+    package_dir = os.path.dirname(importlib.util.find_spec('airflow').origin)
+    directory = os.path.join(package_dir, 'migrations')
+    config = Config(os.path.join(package_dir, 'alembic.ini'))
+    config.set_main_option('script_location', directory)
+    config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN)
+    script_ = ScriptDirectory.from_config(config)
+    with settings.engine.connect() as connection:
+        context = MigrationContext.configure(connection)
+        ticker = 0
+        while True:
+            source_heads = set(script_.get_heads())
+            db_heads = set(context.get_current_heads())
+            if source_heads == db_heads:
+                break
+            if ticker >= timeout:
+                raise TimeoutError("There are still unapplied migrations after {} "
+                                   "seconds.".format(ticker))
+            ticker += 1
+            time.sleep(1)
+            logging.info('Waiting for migrations... %s second(s)', ticker)
 
 Review comment:
   ```suggestion
               log.info('Waiting for migrations... %s second(s)', ticker)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r405785022
 
 

 ##########
 File path: clean-logs.sh
 ##########
 @@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+DIRECTORY=${AIRFLOW_HOME:-/usr/local/airflow}
+RETENTION=${ASTRONOMER__AIRFLOW__WORKER_LOG_RETENTION_DAYS:-15}
 
 Review comment:
   Oh you added a COPY command, missed that.
   
   Still, should probably move this somewhere else in the repo, say scripts/

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#issuecomment-613851298
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=h1) Report
   > Merging [#8219](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b198a1fa94c44228dc7358552aeb6a5371ae0da2&el=desc) will **decrease** coverage by `0.62%`.
   > The diff coverage is `27.58%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8219/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #8219      +/-   ##
   ==========================================
   - Coverage   88.45%   87.83%   -0.63%     
   ==========================================
     Files         940      940              
     Lines       45354    45382      +28     
   ==========================================
   - Hits        40120    39860     -260     
   - Misses       5234     5522     +288     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `84.45% <19.23%> (-13.92%)` | :arrow_down: |
   | [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.29% <100.00%> (+0.01%)` | :arrow_up: |
   | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `95.45% <100.00%> (+0.21%)` | :arrow_up: |
   | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
   | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
   | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
   | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
   | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=footer). Last update [b198a1f...8a08498](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408670949
 
 

 ##########
 File path: airflow/cli/commands/db_command.py
 ##########
 @@ -50,6 +50,16 @@ def upgradedb(args):
     db.upgradedb()
 
 
+@cli_utils.action_logging
 
 Review comment:
   This decorator writes a row to the (audit) log table -- and there is a good chance that this table won't exist when this command is run, so we shouldn't have this decorator here.
   
   ```suggestion
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408690507
 
 

 ##########
 File path: airflow/cli/cli_parser.py
 ##########
 @@ -434,6 +434,12 @@ def __init__(self, flags=None, help=None, action=None, default=None, nargs=None,
 ARG_CFG_PATH = Arg(
     ("--cfg-path",),
     help="Path to config file to use instead of airflow.cfg")
+ARG_MIGRATION_TIMEOUT = Arg(
+    ("-t", "--migration-wait-timeout"),
+    help="timeout to wait for db to migrate ",
+    type=int,
+    default="0",
 
 Review comment:
   We already have `airflow db check`, so perhaps `airflow db check-migrations`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408674357
 
 

 ##########
 File path: scripts/include/clean-logs.sh
 ##########
 @@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
 
 Review comment:
   ```suggestion
   set -euo pipefail
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408671417
 
 

 ##########
 File path: airflow/utils/db.py
 ##########
 @@ -15,9 +15,12 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+import importlib
 import logging
 import os
+import time
 
+from alembic.script import ScriptDirectory
 
 Review comment:
   Please move this in to the wait function -- we don't import this module unless we have to.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408677245
 
 

 ##########
 File path: airflow/cli/cli_parser.py
 ##########
 @@ -434,6 +434,12 @@ def __init__(self, flags=None, help=None, action=None, default=None, nargs=None,
 ARG_CFG_PATH = Arg(
     ("--cfg-path",),
     help="Path to config file to use instead of airflow.cfg")
+ARG_MIGRATION_TIMEOUT = Arg(
+    ("-t", "--migration-wait-timeout"),
+    help="timeout to wait for db to migrate ",
+    type=int,
+    default="0",
 
 Review comment:
   Given the default is 0 (i.e. don't wait) I think we should rename this command,
   
   Rather than `airflow db wait-for-migrations` how about `airflow db check-migrations`, `airflow db is-fully-migrated`? Not 100% happy with either of those so if you have any better ideas?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408671901
 
 

 ##########
 File path: airflow/utils/db.py
 ##########
 @@ -33,9 +36,7 @@
 # noinspection PyUnresolvedReferences
 from airflow.models.serialized_dag import SerializedDagModel  # noqa: F401  # pylint: disable=unused-import
 # TODO: remove create_session once we decide to break backward compatibility
-from airflow.utils.session import (  # noqa: F401 # pylint: disable=unused-import
-    create_session, provide_session,
 
 Review comment:
   This is here because it is part of the API consumed by our users. It needs to stay. (that's why it has the noqa)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r405784605
 
 

 ##########
 File path: clean-logs.sh
 ##########
 @@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+DIRECTORY=${AIRFLOW_HOME:-/usr/local/airflow}
+RETENTION=${ASTRONOMER__AIRFLOW__WORKER_LOG_RETENTION_DAYS:-15}
 
 Review comment:
   ```suggestion
   RETENTION=${AIRFLOW__LOG_RETENTION_DAYS:-15}
   ```
   
   This file won't be included in the apache Docker image without more work being done (cos the prod images do `pip install apache-airflow==...`) so putting this file here won't help/do much by itself.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408674582
 
 

 ##########
 File path: scripts/include/clean-logs.sh
 ##########
 @@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+DIRECTORY=${AIRFLOW_HOME:-/usr/local/airflow}
 
 Review comment:
   This file has lots of shellcheck warnings that we should fix
   ```suggestion
   DIRECTORY="${AIRFLOW_HOME:-/usr/local/airflow}"
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408672173
 
 

 ##########
 File path: airflow/utils/db.py
 ##########
 @@ -561,6 +562,37 @@ def initdb():
     Base.metadata.create_all(settings.engine)  # pylint: disable=no-member
 
 
+def wait_for_migrations(timeout):
+    """
+    Function to wait for all airflow migrations to complete. Used for launching airflow in k8s
 
 Review comment:
   ```suggestion
       Function to wait for all airflow migrations to complete.
   ```
   
   It is nothing specific to k8s

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#issuecomment-613851298
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=h1) Report
   > Merging [#8219](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b198a1fa94c44228dc7358552aeb6a5371ae0da2&el=desc) will **decrease** coverage by `0.62%`.
   > The diff coverage is `27.58%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8219/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #8219      +/-   ##
   ==========================================
   - Coverage   88.45%   87.83%   -0.63%     
   ==========================================
     Files         940      940              
     Lines       45354    45382      +28     
   ==========================================
   - Hits        40120    39860     -260     
   - Misses       5234     5522     +288     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `84.45% <19.23%> (-13.92%)` | :arrow_down: |
   | [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.29% <100.00%> (+0.01%)` | :arrow_up: |
   | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `95.45% <100.00%> (+0.21%)` | :arrow_up: |
   | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
   | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
   | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
   | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
   | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=footer). Last update [b198a1f...8a08498](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] turbaszek commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r409297075
 
 

 ##########
 File path: airflow/cli/cli_parser.py
 ##########
 @@ -952,6 +958,12 @@ class GroupCommand(NamedTuple):
         func=lazy_load_command('airflow.cli.commands.db_command.initdb'),
         args=(),
     ),
+    ActionCommand(
+        name="check-migrations",
+        help="Check if migration have finished (or until continually check until timeout)",
 
 Review comment:
   ```suggestion
           help="Check if migration have finished (or continually check until timeout)",
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408673510
 
 

 ##########
 File path: airflow/utils/db.py
 ##########
 @@ -561,6 +562,37 @@ def initdb():
     Base.metadata.create_all(settings.engine)  # pylint: disable=no-member
 
 
+def wait_for_migrations(timeout):
+    """
+    Function to wait for all airflow migrations to complete. Used for launching airflow in k8s
+    @param timeout:
+    @return:
+    """
+    from alembic.config import Config
+    from alembic.runtime.migration import MigrationContext
+
+    package_dir = os.path.dirname(importlib.util.find_spec('airflow').origin)
+    directory = os.path.join(package_dir, 'migrations')
+    config = Config(os.path.join(package_dir, 'alembic.ini'))
+    config.set_main_option('script_location', directory)
+    config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN)
+    script_ = ScriptDirectory.from_config(config)
 
 Review comment:
   This shares a lot of code with the "setup" un upgradedb (and also that has some bug fixes that this doesn't.)
   
   We should extract that to a new `_get_alembic_config` function.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#issuecomment-613851298
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=h1) Report
   > Merging [#8219](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fcfee821aaeabb9d1e416c1a7dbc552db2f2fd52&el=desc) will **decrease** coverage by `54.98%`.
   > The diff coverage is `37.93%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/8219/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #8219       +/-   ##
   ===========================================
   - Coverage   88.43%   33.45%   -54.99%     
   ===========================================
     Files         940      940               
     Lines       45354    45381       +27     
   ===========================================
   - Hits        40109    15182    -24927     
   - Misses       5245    30199    +24954     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `87.07% <34.61%> (-11.30%)` | :arrow_down: |
   | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `31.81% <50.00%> (-63.42%)` | :arrow_down: |
   | [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `96.39% <100.00%> (-0.89%)` | :arrow_down: |
   | [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [813 more](https://codecov.io/gh/apache/airflow/pull/8219/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=footer). Last update [fcfee82...58743e0](https://codecov.io/gh/apache/airflow/pull/8219?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408675007
 
 

 ##########
 File path: setup.py
 ##########
 @@ -627,6 +627,7 @@ def do_setup():
         entry_points={
             "console_scripts": [
                 "airflow = airflow.__main__:main",
+                "airflow-migration-spinner = airflow.include.airflow_migration_spinner:main",
 
 Review comment:
   Not needed or wanted.
   ```suggestion
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman merged pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
dimberman merged pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8219: Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408674194
 
 

 ##########
 File path: requirements/requirements-python3.6.txt
 ##########
 @@ -123,21 +123,21 @@ future-fstrings==1.2.0
 future==0.18.2
 gcsfs==0.6.1
 google-ads==4.0.0
-google-api-core==1.16.0
+google-api-core==1.17.0
 google-api-python-client==1.8.0
 google-auth-httplib2==0.0.3
 google-auth-oauthlib==0.4.1
-google-auth==1.13.1
+google-auth==1.14.0
 google-cloud-automl==0.10.0
 google-cloud-bigquery-datatransfer==1.0.0
 google-cloud-bigquery==1.24.0
 google-cloud-bigtable==1.2.1
-google-cloud-container==0.4.0
+google-cloud-container==0.5.0
 
 Review comment:
   I wouldn't have expected to see any changes here (and we shouldn't add these changes as part of this PR anyway). I wonder what's caused this though...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] dimberman commented on a change in pull request #8219: [WIP] Add migration waiting script and log cleaner

Posted by GitBox <gi...@apache.org>.
dimberman commented on a change in pull request #8219: [WIP] Add migration waiting script and log cleaner
URL: https://github.com/apache/airflow/pull/8219#discussion_r408270362
 
 

 ##########
 File path: clean-logs.sh
 ##########
 @@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+DIRECTORY=${AIRFLOW_HOME:-/usr/local/airflow}
+RETENTION=${ASTRONOMER__AIRFLOW__WORKER_LOG_RETENTION_DAYS:-15}
 
 Review comment:
   Yeah. was just hacking to get it to work first.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services