You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/17 14:29:18 UTC
[GitHub] [airflow] JavierLopezT opened a new pull request #14853: Improvements on "DAGs and Tasks documentation"
JavierLopezT opened a new pull request #14853:
URL: https://github.com/apache/airflow/pull/14853
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r621534750
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -100,6 +98,11 @@
"""
)
+
Review comment:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596812332
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -89,18 +89,27 @@
# [END basic_task]
# [START documentation]
- dag.doc_md = __doc__
+ with DAG('tutorial',
+ default_args=default_args,
+ schedule_interval=timedelta(days=1)
+ ) as dag:
Review comment:
The purpose of showing only few lines is too highlight the user what the section is about
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596785993
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -89,18 +89,27 @@
# [END basic_task]
# [START documentation]
- dag.doc_md = __doc__
+ with DAG('tutorial',
+ default_args=default_args,
+ schedule_interval=timedelta(days=1)
+ ) as dag:
- t1.doc_md = dedent(
- """\
- #### Task Documentation
- You can document your task using the attributes `doc_md` (markdown),
- `doc` (plain text), `doc_rst`, `doc_json`, `doc_yaml` which gets
- rendered in the UI's Task Instance Details page.
- 
- """
- )
- # [END documentation]
+ t1 = BashOperator(
+ task_id='print_date',
+ bash_command='date',
+ )
+ t1.doc_md = dedent(
Review comment:
The `airflow/example_dags/tutorial.py` should run as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596803383
##########
File path: docs/apache-airflow/tutorial.rst
##########
@@ -184,14 +184,21 @@ Adding DAG and Tasks documentation
----------------------------------
We can add documentation for DAG or each single task. DAG documentation only support
markdown so far and task documentation support plain text, markdown, reStructuredText,
-json, yaml.
+json, yaml. The DAG documentation must be written in a doc string at the beggining of the DAG file to be taken
+in account. Below you can find some examples on how to implement task and DAG docs, as well as screenshots:
Review comment:
Great. Thanks for pointing that out. I'll add this to the documentation as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596607455
##########
File path: docs/apache-airflow/tutorial.rst
##########
@@ -184,14 +184,21 @@ Adding DAG and Tasks documentation
----------------------------------
We can add documentation for DAG or each single task. DAG documentation only support
markdown so far and task documentation support plain text, markdown, reStructuredText,
-json, yaml.
+json, yaml. The DAG documentation must be written in a doc string at the beggining of the DAG file to be taken
+in account. Below you can find some examples on how to implement task and DAG docs, as well as screenshots:
Review comment:
> The DAG documentation must be written in a doc string at the beggining of the DAG file to be taken
in account
This isn't quite true. It just has to be markdown. It only has to be a module docstring if you want to create it by referencing the variable `__doc__`.
You could do something like this anywhere in the file:
```
dag.doc_md = """
# Hello
"""
```
But it is true that using doc_md.__doc__ is a convenient and elegant way to add dag documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#issuecomment-801212303
I have "broken" the tutorial dag file. It was made for being one actual dag split into different sections. However, I find it confusing because in each section there was just one part of the DAG, and in some cases, as the modified, I think it is useful to have the whole picture. So before fixing the static cheks, I would like some feedback on this, just in case I have to make other changes.
Also, I don't know if with the code the SEO can be improved, because if you search DAG documentation or similar in Google it doesn't appear. It doesn't even appear in the search of Airflow docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KulykDmytro commented on pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
KulykDmytro commented on pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#issuecomment-828250655
`task.doc_md` not rendering when
```
AIRFLOW__CORE__STORE_SERIALIZED_DAGS=True
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r621534252
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -100,6 +98,11 @@
"""
)
+
Review comment:
can we use https://user-images.githubusercontent.com/8811558/113521151-7d4d1e00-958f-11eb-9571-10cbc9223193.png for Task Docs ?
since it shows the image on new line.
That image is from my PR: https://github.com/apache/airflow/pull/15191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #14853: Small changes on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #14853:
URL: https://github.com/apache/airflow/pull/14853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596607455
##########
File path: docs/apache-airflow/tutorial.rst
##########
@@ -184,14 +184,21 @@ Adding DAG and Tasks documentation
----------------------------------
We can add documentation for DAG or each single task. DAG documentation only support
markdown so far and task documentation support plain text, markdown, reStructuredText,
-json, yaml.
+json, yaml. The DAG documentation must be written in a doc string at the beggining of the DAG file to be taken
+in account. Below you can find some examples on how to implement task and DAG docs, as well as screenshots:
Review comment:
> The DAG documentation must be written in a doc string at the beggining of the DAG file to be taken
in account
This isn't quite true. It just has to be markdown. It only has to be a module docstring if you want to create it by referencing the variable `__doc__`.
You could do something like this anywhere in the file:
```
dag.doc_md = """
# Hello
"""
```
But it is true that using `doc_md.__doc__` is a convenient and elegant way to add dag documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596812090
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -89,18 +89,27 @@
# [END basic_task]
# [START documentation]
- dag.doc_md = __doc__
+ with DAG('tutorial',
+ default_args=default_args,
+ schedule_interval=timedelta(days=1)
+ ) as dag:
Review comment:
And if you click on "View Source" on the blue strip it should show you full file too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#issuecomment-811797840
[The Workflow run](https://github.com/apache/airflow/actions/runs/708106693) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#issuecomment-811034664
[The Workflow run](https://github.com/apache/airflow/actions/runs/704968459) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596802290
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -89,18 +89,27 @@
# [END basic_task]
# [START documentation]
- dag.doc_md = __doc__
+ with DAG('tutorial',
+ default_args=default_args,
+ schedule_interval=timedelta(days=1)
+ ) as dag:
Review comment:
Because of this: https://github.com/apache/airflow/pull/14853#issuecomment-801212303
If you go directly to the section, you see this
Seeing that, you can wonder where those lines go, so I think it would be more useful to have those lines in the context of an entire DAG. If the tutorial dag must run fine, I think it would be better to include the code as text or in some other file. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596784670
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -89,18 +89,27 @@
# [END basic_task]
# [START documentation]
- dag.doc_md = __doc__
+ with DAG('tutorial',
+ default_args=default_args,
+ schedule_interval=timedelta(days=1)
+ ) as dag:
Review comment:
This isn't right !
We are already in the DAG context manager, why create it again ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596802290
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -89,18 +89,27 @@
# [END basic_task]
# [START documentation]
- dag.doc_md = __doc__
+ with DAG('tutorial',
+ default_args=default_args,
+ schedule_interval=timedelta(days=1)
+ ) as dag:
Review comment:
Because of this: https://github.com/apache/airflow/pull/14853#issuecomment-801212303
If you go directly to the section, you see this
And I think it would be more useful to have those lines in the context of an entire DAG. So if the tutorial dag must run fine, I think it would be better to include the code as text or in some other file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KulykDmytro edited a comment on pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
KulykDmytro edited a comment on pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#issuecomment-828250655
`task.doc_md` not rendering when
```
AIRFLOW__CORE__STORE_SERIALIZED_DAGS=True
```
at least on 1.10.15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596811771
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -89,18 +89,27 @@
# [END basic_task]
# [START documentation]
- dag.doc_md = __doc__
+ with DAG('tutorial',
+ default_args=default_args,
+ schedule_interval=timedelta(days=1)
+ ) as dag:
Review comment:
The entire file is already shown at the top of the doc though:
https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html#example-pipeline-definition
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #14853: Improvements on "DAGs and Tasks documentation"
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #14853:
URL: https://github.com/apache/airflow/pull/14853#discussion_r596785065
##########
File path: airflow/example_dags/tutorial.py
##########
@@ -89,18 +89,27 @@
# [END basic_task]
# [START documentation]
- dag.doc_md = __doc__
+ with DAG('tutorial',
+ default_args=default_args,
+ schedule_interval=timedelta(days=1)
+ ) as dag:
- t1.doc_md = dedent(
- """\
- #### Task Documentation
- You can document your task using the attributes `doc_md` (markdown),
- `doc` (plain text), `doc_rst`, `doc_json`, `doc_yaml` which gets
- rendered in the UI's Task Instance Details page.
- 
- """
- )
- # [END documentation]
+ t1 = BashOperator(
+ task_id='print_date',
+ bash_command='date',
+ )
+ t1.doc_md = dedent(
Review comment:
Same here ! Duplicate of task above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org