You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Sharadh Krishnamurthy <sh...@gmail.com> on 2019/01/03 22:57:21 UTC
Adding more documentation
Hi,
I'm trying to understand airflow internals as part of figuring out things
that are easy, and others that are difficult. Think of it as me reading
thro' these two links:
- https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls
- https://gtoonstra.github.io/etl-with-airflow/gotchas.html
...and understanding *why* those gotchas exist.
As part of this, I realize I am documenting my understanding of the high
level "boxes" in the system (e.g `scheduler`), entities (e.g `DagRun`,
`DagBag`), etc. I am wondering if the committers see value in formally
adding this back as documentation. If yes, are there any guidelines /
prior-art for how to do so?
- Only other prior art of the nature I'm thinking of is
https://cwiki.apache.org/confluence/display/AIRFLOW/Scheduler+Basics
- In my company we're partial to PlantUML (
http://plantuml.com/sequence-diagram) and markdown/rtf. However,
Confluence is does come with plugins for these so that's a possible
alternative.
Excited to hopefully give back to the community.
Best,
Sharadh
Re: Adding more documentation
Posted by Sharadh Krishnamurthy <sh...@gmail.com>.
Very cool, thank you for pointing me to the right place.
On Fri, Jan 4, 2019 at 12:04 PM Matt Davis <ji...@gmail.com> wrote:
> Hi Sharadh,
>
> The main Airflow documentation is here:
> https://airflow.apache.org/index.html
>
> That's built from the source here using Sphinx, and I'm sure contributions
> are welcome: https://github.com/apache/incubator-airflow/tree/master/docs
>
> Best,
> Matt
>
> On Thu, Jan 3, 2019 at 2:57 PM Sharadh Krishnamurthy <
> sharadhk.murthy@gmail.com> wrote:
>
> > Hi,
> >
> > I'm trying to understand airflow internals as part of figuring out things
> > that are easy, and others that are difficult. Think of it as me reading
> > thro' these two links:
> >
> > - https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls
> > - https://gtoonstra.github.io/etl-with-airflow/gotchas.html
> >
> > ...and understanding *why* those gotchas exist.
> >
> > As part of this, I realize I am documenting my understanding of the high
> > level "boxes" in the system (e.g `scheduler`), entities (e.g `DagRun`,
> > `DagBag`), etc. I am wondering if the committers see value in formally
> > adding this back as documentation. If yes, are there any guidelines /
> > prior-art for how to do so?
> >
> > - Only other prior art of the nature I'm thinking of is
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Scheduler+Basics
> > - In my company we're partial to PlantUML (
> > http://plantuml.com/sequence-diagram) and markdown/rtf. However,
> > Confluence is does come with plugins for these so that's a possible
> > alternative.
> >
> > Excited to hopefully give back to the community.
> >
> > Best,
> > Sharadh
> >
>
Re: Adding more documentation
Posted by Matt Davis <ji...@gmail.com>.
Hi Sharadh,
The main Airflow documentation is here:
https://airflow.apache.org/index.html
That's built from the source here using Sphinx, and I'm sure contributions
are welcome: https://github.com/apache/incubator-airflow/tree/master/docs
Best,
Matt
On Thu, Jan 3, 2019 at 2:57 PM Sharadh Krishnamurthy <
sharadhk.murthy@gmail.com> wrote:
> Hi,
>
> I'm trying to understand airflow internals as part of figuring out things
> that are easy, and others that are difficult. Think of it as me reading
> thro' these two links:
>
> - https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls
> - https://gtoonstra.github.io/etl-with-airflow/gotchas.html
>
> ...and understanding *why* those gotchas exist.
>
> As part of this, I realize I am documenting my understanding of the high
> level "boxes" in the system (e.g `scheduler`), entities (e.g `DagRun`,
> `DagBag`), etc. I am wondering if the committers see value in formally
> adding this back as documentation. If yes, are there any guidelines /
> prior-art for how to do so?
>
> - Only other prior art of the nature I'm thinking of is
> https://cwiki.apache.org/confluence/display/AIRFLOW/Scheduler+Basics
> - In my company we're partial to PlantUML (
> http://plantuml.com/sequence-diagram) and markdown/rtf. However,
> Confluence is does come with plugins for these so that's a possible
> alternative.
>
> Excited to hopefully give back to the community.
>
> Best,
> Sharadh
>