You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Tim Swast <sw...@google.com.INVALID> on 2018/05/21 20:35:27 UTC
Where to put docs on configuring specific kinds of connections? (Or
restructuring the docs the way Django does)
Hey folks,
I'd like to write some docs on how to create a GCP connection (and leave
room for documenting other kinds of connections as well). Currently it
seems like there are a couple places such a thing could fit:
- https://airflow.incubator.apache.org/configuration.html#connections
-
https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform
There's also the concepts guide, but I definitely don't think that's the
right place for documenting a specific task like this.
There's a principle I'm used to following with the GCP docs, that the distinct
kinds of documentation
<http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/>
should be organized separately. The Django project does this
https://docs.djangoproject.com/en/2.0/ by splitting into
- Tutorials
- Topic guides (what Airflow calls Concepts)
- Reference guides
- How-to guides
I'd like to propose we split some of the existing "Configuration" topics
into separate how-to guides. What do you think?
Meta: Should I create JIRA issues for this kind of pre-discussion or start
here as I've done?
* • **Tim Swast*
* • *Software Friendliness Engineer
* • *Google Cloud Developer Relations
* • *Seattle, WA, USA
Re: Where to put docs on configuring specific kinds of connections?
(Or restructuring the docs the way Django does)
Posted by Tim Swast <sw...@google.com.INVALID>.
PR: https://github.com/apache/incubator-airflow/pull/3400
Jira: https://issues.apache.org/jira/browse/AIRFLOW-2509
On Tue, May 22, 2018 at 9:12 AM Tim Swast <sw...@google.com> wrote:
> I think the whole "Configuration" page could benefit from getting split
> into separate how-to guides. Basically, every sub-heading in
> Configuration becomes a different how-to guide. Since the configuration
> page is attempting to do double-duty as a tutorial right now, keep the
> current sequence, at least until a "Deploying a Production Airflow
> Environment" tutorial is written.
>
> I started this split (PR coming soon). I think the split document makes it
> clearer where some additional guidance would be useful:
>
> - Connections: could use some specific examples for how to created
> different kinds of connections.
> - Database backends: could use some specific examples on configuring a
> MySQL or Postgres database backend.
>
> * • **Tim Swast*
> * • *Software Friendliness Engineer
> * • *Google Cloud Developer Relations
> * • *Seattle, WA, USA
>
>
> On Mon, May 21, 2018 at 10:20 PM Taylor Edmiston <te...@gmail.com>
> wrote:
>
>> Hey Tim -
>>
>> I came to Airflow from the Django world as well and had the same thought
>> that much of the work that's been put into their docs over time could be
>> applied here too. In terms of documentation for large Python projects,
>> perhaps they're the gold standard.
>>
>> Can you give a few examples of the specific docs you think would benefit
>> from refactoring out here? I'd be happy to assist with this effort as well.
>>
>> Taylor
>> Sent from my iPhone
>>
>> > On May 21, 2018, at 4:35 PM, Tim Swast <sw...@google.com.INVALID>
>> wrote:
>> >
>> > Hey folks,
>> >
>> > I'd like to write some docs on how to create a GCP connection (and leave
>> > room for documenting other kinds of connections as well). Currently it
>> > seems like there are a couple places such a thing could fit:
>> >
>> > - https://airflow.incubator.apache.org/configuration.html#connections
>> > -
>> >
>> https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform
>> >
>> > There's also the concepts guide, but I definitely don't think that's the
>> > right place for documenting a specific task like this.
>> >
>> > There's a principle I'm used to following with the GCP docs, that the
>> distinct
>> > kinds of documentation
>> > <
>> http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/
>> >
>> > should be organized separately. The Django project does this
>> > https://docs.djangoproject.com/en/2.0/ by splitting into
>> >
>> > - Tutorials
>> > - Topic guides (what Airflow calls Concepts)
>> > - Reference guides
>> > - How-to guides
>> >
>> > I'd like to propose we split some of the existing "Configuration" topics
>> > into separate how-to guides. What do you think?
>> >
>> > Meta: Should I create JIRA issues for this kind of pre-discussion or
>> start
>> > here as I've done?
>> >
>> > * • **Tim Swast*
>> > * • *Software Friendliness Engineer
>> > * • *Google Cloud Developer Relations
>> > * • *Seattle, WA, USA
>>
>
Re: Where to put docs on configuring specific kinds of connections?
(Or restructuring the docs the way Django does)
Posted by Tim Swast <sw...@google.com.INVALID>.
I think the whole "Configuration" page could benefit from getting split
into separate how-to guides. Basically, every sub-heading in Configuration
becomes a different how-to guide. Since the configuration page is
attempting to do double-duty as a tutorial right now, keep the current
sequence, at least until a "Deploying a Production Airflow Environment"
tutorial is written.
I started this split (PR coming soon). I think the split document makes it
clearer where some additional guidance would be useful:
- Connections: could use some specific examples for how to created
different kinds of connections.
- Database backends: could use some specific examples on configuring a
MySQL or Postgres database backend.
* • **Tim Swast*
* • *Software Friendliness Engineer
* • *Google Cloud Developer Relations
* • *Seattle, WA, USA
On Mon, May 21, 2018 at 10:20 PM Taylor Edmiston <te...@gmail.com>
wrote:
> Hey Tim -
>
> I came to Airflow from the Django world as well and had the same thought
> that much of the work that's been put into their docs over time could be
> applied here too. In terms of documentation for large Python projects,
> perhaps they're the gold standard.
>
> Can you give a few examples of the specific docs you think would benefit
> from refactoring out here? I'd be happy to assist with this effort as well.
>
> Taylor
> Sent from my iPhone
>
> > On May 21, 2018, at 4:35 PM, Tim Swast <sw...@google.com.INVALID> wrote:
> >
> > Hey folks,
> >
> > I'd like to write some docs on how to create a GCP connection (and leave
> > room for documenting other kinds of connections as well). Currently it
> > seems like there are a couple places such a thing could fit:
> >
> > - https://airflow.incubator.apache.org/configuration.html#connections
> > -
> >
> https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform
> >
> > There's also the concepts guide, but I definitely don't think that's the
> > right place for documenting a specific task like this.
> >
> > There's a principle I'm used to following with the GCP docs, that the
> distinct
> > kinds of documentation
> > <
> http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/
> >
> > should be organized separately. The Django project does this
> > https://docs.djangoproject.com/en/2.0/ by splitting into
> >
> > - Tutorials
> > - Topic guides (what Airflow calls Concepts)
> > - Reference guides
> > - How-to guides
> >
> > I'd like to propose we split some of the existing "Configuration" topics
> > into separate how-to guides. What do you think?
> >
> > Meta: Should I create JIRA issues for this kind of pre-discussion or
> start
> > here as I've done?
> >
> > * • **Tim Swast*
> > * • *Software Friendliness Engineer
> > * • *Google Cloud Developer Relations
> > * • *Seattle, WA, USA
>
Re: Where to put docs on configuring specific kinds of connections? (Or restructuring the docs the way Django does)
Posted by Taylor Edmiston <te...@gmail.com>.
Hey Tim -
I came to Airflow from the Django world as well and had the same thought that much of the work that's been put into their docs over time could be applied here too. In terms of documentation for large Python projects, perhaps they're the gold standard.
Can you give a few examples of the specific docs you think would benefit from refactoring out here? I'd be happy to assist with this effort as well.
Taylor
Sent from my iPhone
> On May 21, 2018, at 4:35 PM, Tim Swast <sw...@google.com.INVALID> wrote:
>
> Hey folks,
>
> I'd like to write some docs on how to create a GCP connection (and leave
> room for documenting other kinds of connections as well). Currently it
> seems like there are a couple places such a thing could fit:
>
> - https://airflow.incubator.apache.org/configuration.html#connections
> -
> https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform
>
> There's also the concepts guide, but I definitely don't think that's the
> right place for documenting a specific task like this.
>
> There's a principle I'm used to following with the GCP docs, that the distinct
> kinds of documentation
> <http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/>
> should be organized separately. The Django project does this
> https://docs.djangoproject.com/en/2.0/ by splitting into
>
> - Tutorials
> - Topic guides (what Airflow calls Concepts)
> - Reference guides
> - How-to guides
>
> I'd like to propose we split some of the existing "Configuration" topics
> into separate how-to guides. What do you think?
>
> Meta: Should I create JIRA issues for this kind of pre-discussion or start
> here as I've done?
>
> * • **Tim Swast*
> * • *Software Friendliness Engineer
> * • *Google Cloud Developer Relations
> * • *Seattle, WA, USA
Re: Where to put docs on configuring specific kinds of connections?
(Or restructuring the docs the way Django does)
Posted by Naik Kaxil <k....@reply.com>.
Hi Tim,
My personal opinion is to write the docs on GCP connection down at GCP Integration page <https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform> would be an ideal fit but would like to hear others opinion as well.
+1 for creating separate How-To guides.
Regards,
Kaxil
On 21/05/2018, 21:35, "Tim Swast" <sw...@google.com.INVALID> wrote:
Hey folks,
I'd like to write some docs on how to create a GCP connection (and leave
room for documenting other kinds of connections as well). Currently it
seems like there are a couple places such a thing could fit:
- https://airflow.incubator.apache.org/configuration.html#connections
-
https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform
There's also the concepts guide, but I definitely don't think that's the
right place for documenting a specific task like this.
There's a principle I'm used to following with the GCP docs, that the distinct
kinds of documentation
<http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/>
should be organized separately. The Django project does this
https://docs.djangoproject.com/en/2.0/ by splitting into
- Tutorials
- Topic guides (what Airflow calls Concepts)
- Reference guides
- How-to guides
I'd like to propose we split some of the existing "Configuration" topics
into separate how-to guides. What do you think?
Meta: Should I create JIRA issues for this kind of pre-discussion or start
here as I've done?
* • **Tim Swast*
* • *Software Friendliness Engineer
* • *Google Cloud Developer Relations
* • *Seattle, WA, USA
Kaxil Naik
Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.naik@reply.com
www.reply.com