You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/19 23:14:51 UTC
[GitHub] [airflow] atsalolikhin-spokeo opened a new pull request #10413: Add documentation for preparing database for Airflow
atsalolikhin-spokeo opened a new pull request #10413:
URL: https://github.com/apache/airflow/pull/10413
Closes #10389
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r473428052
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,12 +48,31 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Configure Airflow's database connection string
+----------------------------------------------
+
Once you've setup your database to host Airflow, you'll need to alter the
SqlAlchemy connection string located in your configuration file
``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
setting to use "LocalExecutor", an executor that can parallelize task
instances locally.
Review comment:
```suggestion
Configure Airflow's database connection string
----------------------------------------------
Once you've setup your database to host Airflow, you'll need to alter the
SqlAlchemy connection string located in ``sql_alchemy_conn`` option in ``[core]`` section in your configuration file
``$AIRFLOW_HOME/airflow.cfg``.
Configure a worker that supports parallelism
--------------------------------------------
You should then also change the "executor" option in the ``[core]`` option to use "LocalExecutor", an executor that can parallelize task instances locally.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #10413:
URL: https://github.com/apache/airflow/pull/10413
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
atsalolikhin-spokeo commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r485787872
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow;
+ CREATE USER airflow WITH PASSWORD 'airflow';
+ GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
+
+You may need to update your Postgres ``pg_hba.conf`` to add the
+``airflow`` user to the database access control list; and to reload
+the database configuration to load your change. See
+[The pg_hba.conf File](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html)
+in the Postgres documentation to learn more.
+
+Configure Airflow's database connection string
+----------------------------------------------
+
Once you've setup your database to host Airflow, you'll need to alter the
-SqlAlchemy connection string located in your configuration file
-``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
-setting to use "LocalExecutor", an executor that can parallelize task
-instances locally.
+SqlAlchemy connection string located in ``sql_alchemy_conn`` option in ``[core]`` section in your configuration file
+``$AIRFLOW_HOME/airflow.cfg``.
Review comment:
Okay, thank you very much! :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r480287444
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,49 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8_mb3 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+
+If you would like to use `utf8mb4` encoding, you need to add an additional configuration option `sql_engine_collation_for_ids` set to `utf8mb3_general_ci` - otherwise index sizes of some Airflow tables will exceed maximum index size allowed in MySQL.
Review comment:
```suggestion
If you would like to use ``utf8mb4`` encoding, you need to add an additional configuration option ``sql_engine_collation_for_ids`` set to ``utf8mb3_general_ci`` - otherwise index sizes of some Airflow tables will exceed maximum index size allowed in MySQL.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r479657692
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8_mb3 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
Review comment:
```suggestion
If you would like to use `utf8mb4` encoding, you need to add an additional configuration option `sql_engine_collation_for_ids` set to `utf8mb3_general_ci` - otherwise index sizes of some Airflow tables will exceed maximum index size allowed in MySQL.
```
I still we need to add this :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r473425753
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,12 +48,31 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
Review comment:
Can you also add example for PostgresSQL?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#issuecomment-676803644
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r479543634
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow;
+ CREATE USER airflow WITH PASSWORD 'airflow';
+ GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
+
+You may need to update your Postgres `pg_hba.conf` to add the
+`airflow` user to the database access control list; and to reload
+the database configuration to load your change. See
+[The pg_hba.conf File](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html)
+in the Postgres documentation to learn more.
+
+Configure Airflow's database connection string
+----------------------------------------------
+
Once you've setup your database to host Airflow, you'll need to alter the
-SqlAlchemy connection string located in your configuration file
-``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
-setting to use "LocalExecutor", an executor that can parallelize task
-instances locally.
+SqlAlchemy connection string located in ``sql_alchemy_conn`` option in ``[core]`` section in your configuration file
+``$AIRFLOW_HOME/airflow.cfg``.
+
+Configure a worker that supports parallelism
+--------------------------------------------
+
+You should then also change the "executor" option in the ``[core]`` option to use "LocalExecutor", an executor that can parallelize task instances locally.
Review comment:
```suggestion
You should then also change the ``executor`` option in the ``[core]`` section to ``LocalExecutor``. an executor that can parallelize task instances locally.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
atsalolikhin-spokeo commented on pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#issuecomment-689200729
@mik-laj Thank you for your kind comment and direction. I've fixed the code so the CI checks are happy, and rebased to tip of master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#issuecomment-686817728
@atsalolikhin-spokeo CI checks are sad. Can you fix it? Once you do this, I will be very happy to merge this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
atsalolikhin-spokeo commented on pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#issuecomment-676804295
Yay, thank you! :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r473500806
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow;
+ CREATE USER airflow WITH PASSWORD 'airflow';
+ GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
+
+You may need to update your Postgres `pg_hba.conf` to add the
+`airflow` user to the database access control list; and to reload
+the database configuration to load your change. See
+`The pg_hba.conf File <https://www.postgresql.org/docs/current/auth-pg-hba-conf.html>`__
+in the Postgres documentation to learn more.
+
+Configure Airflow's database connection string
+----------------------------------------------
+
Once you've setup your database to host Airflow, you'll need to alter the
-SqlAlchemy connection string located in your configuration file
-``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
-setting to use "LocalExecutor", an executor that can parallelize task
-instances locally.
+SqlAlchemy connection string located in ``sql_alchemy_conn`` option in ``[core]`` section in your configuration file
+``$AIRFLOW_HOME/airflow.cfg``.
+
+Configure a worker that supports parallelism
+--------------------------------------------
+
+You should then also change the "executor" option in the ``[core]`` option to use "LocalExecutor", an executor that can parallelize task instances locally.
Review comment:
```suggestion
You should then also change the ``executor`` option in the ``[core]`` option to use ``LocalExecutor``, an executor that can parallelize task instances locally.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r485485991
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow;
+ CREATE USER airflow WITH PASSWORD 'airflow';
+ GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
+
+You may need to update your Postgres ``pg_hba.conf`` to add the
+``airflow`` user to the database access control list; and to reload
+the database configuration to load your change. See
+[The pg_hba.conf File](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html)
+in the Postgres documentation to learn more.
+
+Configure Airflow's database connection string
+----------------------------------------------
+
Once you've setup your database to host Airflow, you'll need to alter the
-SqlAlchemy connection string located in your configuration file
-``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
-setting to use "LocalExecutor", an executor that can parallelize task
-instances locally.
+SqlAlchemy connection string located in ``sql_alchemy_conn`` option in ``[core]`` section in your configuration file
+``$AIRFLOW_HOME/airflow.cfg``.
Review comment:
It would be ideal to mention the configuration about environment variables here, because this option is configured this way, but I can do it in a separate PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#issuecomment-689456277
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r473500561
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow;
+ CREATE USER airflow WITH PASSWORD 'airflow';
+ GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
+
+You may need to update your Postgres `pg_hba.conf` to add the
+`airflow` user to the database access control list; and to reload
Review comment:
```suggestion
You may need to update your Postgres ``pg_hba.conf`` to add the
``airflow`` user to the database access control list; and to reload
```
rst required double `.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r480287828
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8_mb3 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
Review comment:
My bad with single backticks :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r473664358
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,48 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
+
+ CREATE DATABASE airflow;
+ CREATE USER airflow WITH PASSWORD 'airflow';
+ GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
+
+You may need to update your Postgres ``pg_hba.conf`` to add the
+``airflow`` user to the database access control list; and to reload
+the database configuration to load your change. See
+`The pg_hba.conf File <https://www.postgresql.org/docs/current/auth-pg-hba-conf.html>`__
+in the Postgres documentation to learn more.
+
+Configure Airflow's database connection string
+----------------------------------------------
+
Once you've setup your database to host Airflow, you'll need to alter the
-SqlAlchemy connection string located in your configuration file
-``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
-setting to use "LocalExecutor", an executor that can parallelize task
-instances locally.
+SqlAlchemy connection string located in ``sql_alchemy_conn`` option in ``[core]`` section in your configuration file
+``$AIRFLOW_HOME/airflow.cfg``.
+
+Configure a worker that supports parallelism
+--------------------------------------------
+
+You should then also change the ``executor`` option in the ``[core]`` option to use ``LocalExecutor``, an executor that can parallelize task instances locally.
Review comment:
Maybe
```suggestion
You should then also change the ``executor`` option in the ``[core]`` option to not use the ``SequentiaExecutor``. Teh ``SequentialExecutor`` cannot parallelize task instance execution. You need to use the `LocalExecutor` if you want to run tasks on single machine or ``CeleryExecutor``, ``KubernetesExecutor`` or ``DaskExecutor`` (those are example ones) if you distribute your tasks between multiple machines.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r473664358
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,48 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
+
+ CREATE DATABASE airflow;
+ CREATE USER airflow WITH PASSWORD 'airflow';
+ GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
+
+You may need to update your Postgres ``pg_hba.conf`` to add the
+``airflow`` user to the database access control list; and to reload
+the database configuration to load your change. See
+`The pg_hba.conf File <https://www.postgresql.org/docs/current/auth-pg-hba-conf.html>`__
+in the Postgres documentation to learn more.
+
+Configure Airflow's database connection string
+----------------------------------------------
+
Once you've setup your database to host Airflow, you'll need to alter the
-SqlAlchemy connection string located in your configuration file
-``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
-setting to use "LocalExecutor", an executor that can parallelize task
-instances locally.
+SqlAlchemy connection string located in ``sql_alchemy_conn`` option in ``[core]`` section in your configuration file
+``$AIRFLOW_HOME/airflow.cfg``.
+
+Configure a worker that supports parallelism
+--------------------------------------------
+
+You should then also change the ``executor`` option in the ``[core]`` option to use ``LocalExecutor``, an executor that can parallelize task instances locally.
Review comment:
Maybe
```suggestion
You should then also change the ``executor`` option in the ``[core]`` option to not use ``SequentiaExecutor``. Sequential executor cannot parallelize task instance execution. You need to use `LocalExecutor` if you want to run tasks on single machine or `CeleryExecutor`, `KubernetesExecutor` or `DaskExecutor` for example if you distribute your tasks).
```
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,48 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
Review comment:
```suggestion
CREATE DATABASE airflow CHARACTER SET utf8_mb3 COLLATE utf8_unicode_ci;
```
Additionally we must add section that describes how to configure Airflow with mysql `utf8_mb4`.
Airflow will not work without additional configuration with `utf8_mb4` character set. The index size in few tables is too large for it (maximum index size in 5.7 and 8.0 is 3072 characters and we are exceeding the limit if we use utf8_mb4 - each character in `utf8mb4` is 4 bytes rather than 3 bytes in `utf8_mb3`): https://dev.mysql.com/doc/refman/8.0/en/innodb-limits.html.
This is true for MySQL 5.7 but also remains a problem in MySQL 8, even if we do not officially support it (yet). Additionally: in MySQL 8 `utf8_mb3` is deprecated and `utf8_mb4` is default when `utf8` is used: https://mysqlserverteam.com/mysql-8-0-when-to-use-utf8mb3-over-utf8mb4/
There is a change I implemented that allows using `utf8mb4` character set but it requires special configuration of Airflow: https://github.com/apache/airflow/pull/7570/files
How it works - you need to set `sql_engine_collation_for_ids` parameter in the configuration of Airflow to `utf8_mb3` if the whole database is `utf8_mb4`. This will switch all the long columns that are used in indexes to use `utf8_mb3`, so the index for those columns does not exceed the maximum index size.
Hopefully, in MySQL 9 (or 10?) they will further increase the index size ¯\_(ツ)_/¯
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
atsalolikhin-spokeo commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r479552845
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,48 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
+
+ CREATE DATABASE airflow;
+ CREATE USER airflow WITH PASSWORD 'airflow';
+ GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
+
+You may need to update your Postgres ``pg_hba.conf`` to add the
+``airflow`` user to the database access control list; and to reload
+the database configuration to load your change. See
+`The pg_hba.conf File <https://www.postgresql.org/docs/current/auth-pg-hba-conf.html>`__
+in the Postgres documentation to learn more.
+
+Configure Airflow's database connection string
+----------------------------------------------
+
Once you've setup your database to host Airflow, you'll need to alter the
-SqlAlchemy connection string located in your configuration file
-``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
-setting to use "LocalExecutor", an executor that can parallelize task
-instances locally.
+SqlAlchemy connection string located in ``sql_alchemy_conn`` option in ``[core]`` section in your configuration file
+``$AIRFLOW_HOME/airflow.cfg``.
+
+Configure a worker that supports parallelism
+--------------------------------------------
+
+You should then also change the ``executor`` option in the ``[core]`` option to use ``LocalExecutor``, an executor that can parallelize task instances locally.
Review comment:
I've taken the suggested edit from @mik-laj , thanks, @potiuk !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r473500291
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow;
+ CREATE USER airflow WITH PASSWORD 'airflow';
+ GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;
+
+You may need to update your Postgres `pg_hba.conf` to add the
+`airflow` user to the database access control list; and to reload
+the database configuration to load your change. See
+[The pg_hba.conf File](https://www.postgresql.org/docs/current/auth-pg-hba-conf.html)
Review comment:
```suggestion
`The pg_hba.conf File <https://www.postgresql.org/docs/current/auth-pg-hba-conf.html>`__
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #10413: Add documentation for preparing database for Airflow
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #10413:
URL: https://github.com/apache/airflow/pull/10413#discussion_r473499999
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
Review comment:
```suggestion
.. code-block:: sql
```
##########
File path: docs/howto/initialize-database.rst
##########
@@ -48,11 +48,46 @@ SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
want to set a default schema for your role with a
command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+Setup your database to host Airflow
+-----------------------------------
+
+Create a database called ``airflow`` and a database user that Airflow
+will use to access this database.
+
+Example, for **MySQL**:
+
+.. code-block:: sql
+ CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
+ CREATE USER 'airflow' IDENTIFIED BY 'airflow';
+ GRANT ALL PRIVILEGES ON airflow.* TO 'airflow';
+
+Example, for **Postgres**:
+
+.. code-block:: sql
Review comment:
```suggestion
.. code-block:: sql
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org