You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/17 19:30:40 UTC

[GitHub] [airflow] shicholas opened a new issue #22349: 2022 Summer of Docs Proposal

shicholas opened a new issue #22349:
URL: https://github.com/apache/airflow/issues/22349

### Description

From email discussions on the Apache listserv, I'd like to propose the following Google Summer of Docs program for this wonderful project. These are just a first take and each of the sections should warrant its own deep-dive discussion (maybe on a GitHub issue or discussion). I think each section can benefit from interactive visuals which we can include as part of the summer of docs.

## Airflow with Your Team
Working together on complex software engineering is never an easy task. These docs contain conventions and suggestions on how you can effectively work together on your team.

### Git-Ops
We recommend that teams work together on an Airflow project with `git`. Teams should yearn to have one and only one `main` branch which reflects what is deployed to their production environment. Every change into the `main` branch should happen via a pull request so it's tested and reviewed.

### Continuous Integration
We recommend automatically enforcing as many rules as possible by using tools like [GitHub's Super Linter](https://github.com/github/super-linter). For Python projects, this means linting your code with Black, isort and Flake8 (the same linters required to publish code here).

We also recommend running as many tests as possible to test the arity of your methods and the accuracy of your SQL scripts.

### Staging Environment
For many teams, a true staging environment is not feasible because data constantly changes and data warehouses are very large.
_I'm really not sure what people's takes are here, I think a staging environment isn't worth the effort and instead running DAGs with the built-in feature flagging is the best way to maintain velocity_.

### Hotfixes
Instead of a staging environment, your team may consider a hotfix environment which has parity with production and additionally contains hotfix changes that need to be deployed immediately. We recommend deploying the hot fix server from pull requests.

### Use case/motivation

I do DevOps for a data engineering team heavily reliant on Airflow to manage our scripting. We have iterated several times on how we work together with Airflow and would like to collaborate with other teams on how they best use Airflow to improve our process.

I think the issues proposed above are common problems many teams face, and having helpful documentation will point us all towards how to use Airflow amongst teams.

### Related issues

None that I know of that specifically call out working as a team.

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org