You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/19 21:42:39 UTC

[GitHub] [airflow] francescomucio opened a new pull request #11029: Starting breeze will run an init script after the environment is setup

francescomucio opened a new pull request #11029:
URL: https://github.com/apache/airflow/pull/11029


   Starting automatically Airflow with --start-airflow made me realize the necessity to run some additional commands to complete the setup of my development or testing environment: using variables.env was not nice, but also it was wrong, because that file is sourced before --start-airflow initialize the db (so no way to create a connection or a pool before the database is up).
   
   For these reasons I gave Airflow the possibility to run an init script (`init.sh`) from `files/airflow-breeze-config`. The flag `--init-script` is added as commodity to change the name of the script (the folder should stay the same).
   
   The init script is executed after the environment is setup, but before other command (like the `exec` command or tests). The only exception is the `--start-airflow` flag, in this case the script is executed after the db initialization, but before the tmux session is created.
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] francescomucio commented on pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
francescomucio commented on pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#issuecomment-695755551


   not sure why pylint is still failing


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#issuecomment-695826089


   Ah no. The error was quite clear:
   
   ```
   In scripts/in_container/run_init_script.sh line 26:
   if [ -z ${INIT_SCRIPT_FILE} ]; then
           ^-----------------^ SC2086: Double quote to prevent globbing and word splitting.
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] francescomucio edited a comment on pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
francescomucio edited a comment on pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#issuecomment-695845128


   @potiuk I need to learn to read these error messages. New commit fixed it


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#discussion_r491953483



##########
File path: breeze-complete
##########
@@ -138,7 +138,7 @@ _breeze_long_options="
 help python: backend: integration:
 kubernetes-mode: kubernetes-version: helm-version: kind-version:
 skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
-verbose assume-yes assume-no assume-quit forward-credentials start-airflow
+verbose assume-yes assume-no assume-quit forward-credentials start-airflow init-script:

Review comment:
       Oh yes, you got it right




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] francescomucio edited a comment on pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
francescomucio edited a comment on pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#issuecomment-695845128


   @potiuk I need to learn to read these error messages. New commit fixed it


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#discussion_r491608793



##########
File path: BREEZE.rst
##########
@@ -1957,6 +1957,11 @@ This is the current syntax for  `./breeze <./breeze>`_:
           available for the usual use cases. In case the database is not initialized it will
           run also 'airflow db init' and create an Admin user with credentials admin/admin.
 
+  --init-script <INIT_SCRIPT_FILE>
+          Initialization script name - Sourced from files/airflow-breeze-config. Default value
+          init.sh. I will be executed after the environment is configured and started (in case

Review comment:
       ```suggestion
             init.sh. It will be executed after the environment is configured and started (in case
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] francescomucio commented on a change in pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
francescomucio commented on a change in pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#discussion_r491947440



##########
File path: breeze-complete
##########
@@ -138,7 +138,7 @@ _breeze_long_options="
 help python: backend: integration:
 kubernetes-mode: kubernetes-version: helm-version: kind-version:
 skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
-verbose assume-yes assume-no assume-quit forward-credentials start-airflow
+verbose assume-yes assume-no assume-quit forward-credentials start-airflow init-script:

Review comment:
       I assumed the colon was for flags accepting a parameter, --init-script needs the name of the file. Did I got it wrong?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil merged pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #11029:
URL: https://github.com/apache/airflow/pull/11029


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil merged pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #11029:
URL: https://github.com/apache/airflow/pull/11029


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#discussion_r491608842



##########
File path: breeze
##########
@@ -1966,6 +1972,11 @@ ${FORMATTED_INTEGRATIONS}
         available for the usual use cases. In case the database is not initialized it will
         run also 'airflow db init' and create an Admin user with credentials admin/admin.
 
+--init-script <INIT_SCRIPT_FILE>
+        Initialization script name - Sourced from files/airflow-breeze-config. Default value
+        init.sh. I will be executed after the environment is configured and started (in case

Review comment:
       ```suggestion
           init.sh. It will be executed after the environment is configured and started (in case
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#issuecomment-695826089


   Ah no. The error was quite clear:
   
   ```
   In scripts/in_container/run_init_script.sh line 26:
   if [ -z ${INIT_SCRIPT_FILE} ]; then
           ^-----------------^ SC2086: Double quote to prevent globbing and word splitting.
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] francescomucio commented on pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
francescomucio commented on pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#issuecomment-695845128


   @potiuk I need to learn to read these error messages. New commit coming soon.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#discussion_r491942690



##########
File path: breeze-complete
##########
@@ -138,7 +138,7 @@ _breeze_long_options="
 help python: backend: integration:
 kubernetes-mode: kubernetes-version: helm-version: kind-version:
 skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
-verbose assume-yes assume-no assume-quit forward-credentials start-airflow
+verbose assume-yes assume-no assume-quit forward-credentials start-airflow init-script:

Review comment:
       is the trailing ":" (colon) here intended?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] francescomucio commented on pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
francescomucio commented on pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#issuecomment-695845128


   @potiuk I need to learn to read these error messages. New commit coming soon.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#discussion_r491942690



##########
File path: breeze-complete
##########
@@ -138,7 +138,7 @@ _breeze_long_options="
 help python: backend: integration:
 kubernetes-mode: kubernetes-version: helm-version: kind-version:
 skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
-verbose assume-yes assume-no assume-quit forward-credentials start-airflow
+verbose assume-yes assume-no assume-quit forward-credentials start-airflow init-script:

Review comment:
       is the trailing ":" (colon) here intended?

##########
File path: breeze-complete
##########
@@ -138,7 +138,7 @@ _breeze_long_options="
 help python: backend: integration:
 kubernetes-mode: kubernetes-version: helm-version: kind-version:
 skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
-verbose assume-yes assume-no assume-quit forward-credentials start-airflow
+verbose assume-yes assume-no assume-quit forward-credentials start-airflow init-script:

Review comment:
       Oh yes, you got it right




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] francescomucio commented on a change in pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
francescomucio commented on a change in pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#discussion_r491947440



##########
File path: breeze-complete
##########
@@ -138,7 +138,7 @@ _breeze_long_options="
 help python: backend: integration:
 kubernetes-mode: kubernetes-version: helm-version: kind-version:
 skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
-verbose assume-yes assume-no assume-quit forward-credentials start-airflow
+verbose assume-yes assume-no assume-quit forward-credentials start-airflow init-script:

Review comment:
       I assumed the colon was for flags accepting a parameter, --init-script needs the name of the file. Did I got it wrong?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] francescomucio commented on a change in pull request #11029: Starting breeze will run an init script after the environment is setup

Posted by GitBox <gi...@apache.org>.
francescomucio commented on a change in pull request #11029:
URL: https://github.com/apache/airflow/pull/11029#discussion_r491663285



##########
File path: breeze
##########
@@ -1966,6 +1972,11 @@ ${FORMATTED_INTEGRATIONS}
         available for the usual use cases. In case the database is not initialized it will
         run also 'airflow db init' and create an Admin user with credentials admin/admin.
 
+--init-script <INIT_SCRIPT_FILE>
+        Initialization script name - Sourced from files/airflow-breeze-config. Default value
+        init.sh. I will be executed after the environment is configured and started (in case

Review comment:
       fixed
   

##########
File path: BREEZE.rst
##########
@@ -1957,6 +1957,11 @@ This is the current syntax for  `./breeze <./breeze>`_:
           available for the usual use cases. In case the database is not initialized it will
           run also 'airflow db init' and create an Admin user with credentials admin/admin.
 
+  --init-script <INIT_SCRIPT_FILE>
+          Initialization script name - Sourced from files/airflow-breeze-config. Default value
+          init.sh. I will be executed after the environment is configured and started (in case

Review comment:
       fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org