You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/29 23:36:23 UTC
[GitHub] [airflow] MatrixManAtYrService opened a new pull request #20567: params concept doc rework
MatrixManAtYrService opened a new pull request #20567:
URL: https://github.com/apache/airflow/pull/20567
I think that this doc could be improved by adding examples of how to reference the params in your dag. (Also, the current example code causes this: https://github.com/apache/airflow/issues/20559.)
While trying to find the right place to work a few reference examples in, I ended up rewriting quite a lot of it.
Let me know if you think that this is an improvement.
I haven't yet figured out how to build this and view it locally, and I'd want to do that as a sanity check before merging it, but I figured get feedback on what I've written before I do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #20567: Improve documentation on ``Params``
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #20567:
URL: https://github.com/apache/airflow/pull/20567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1002815759
> I haven't yet figured out how to build this and view it locally, and I'd want to do that as a sanity check before merging it, but I figured get feedback on what I've written before I do that.
`./breeze build-docs -- --package-filter apache-airlfow`
Then `./docs/start_doc_server.sh`
You will also see result of building the docs here in CI .
More info here: https://github.com/apache/airflow/blob/main/docs/README.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#discussion_r776812156
##########
File path: docs/apache-airflow/concepts/params.rst
##########
@@ -18,46 +18,115 @@
Params
======
-Params are Airflow's concept of providing runtime configuration to tasks when a DAG gets triggered manually.
-Params are configured while defining the DAG & tasks, that can be altered while doing a manual trigger. The
-ability to update params while triggering a DAG depends on the flag ``core.dag_run_conf_overrides_params``,
-so if that flag is ``False``, params would behave like constants.
+Params are Airflow's way to provide runtime configuration to tasks when a DAG gets triggered manually.
+To use them, initialize your DAG with a dictionary where the keys are strings with each param's name, and the values are ``Param`` objects.
-To use them, one can use the ``Param`` class for complex trigger-time validations or simply use primitive types,
-which won't be doing any such validations.
+``Param`` makes use of `json-schema <https://json-schema.org/>`, so one can use the full json-schema specifications mentioned at https://json-schema.org/draft/2020-12/json-schema-validation.html to define the construct of a ``Param`` objects.
+Or, if you want a default value without any validation, you can use literals instead.
.. code-block::
+ :caption a simple DAG with a parameter
+ from airflow import DAG
+ from airflow.models.param import Param
+ from airflow.operators.python_operator import PythonOperator
+
+ with DAG(
+ "params",
+ params={"x": Param(5, type="integer", minimum=3),
+ "y": 6},
+ ) as the_dag:
+
+ def print_x(**context):
+ print(context["params"]["x"])
+
+ # prints 5, or whatever the user provided at trigger time
+ PythonOperator(
+ task_id="print_x",
+ python_callable=print_it,
+ )
+
+Params can also be added to individual tasks.
+If there's already a dag param with that name, the task-level default will take precedence over the dag-level default.
+
+.. code-block::
+ :caption tasks can have parameters too
Review comment:
Same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1004490644
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1002815759
> I haven't yet figured out how to build this and view it locally, and I'd want to do that as a sanity check before merging it, but I figured get feedback on what I've written before I do that.
`./breeze build-docs -- --package-filter apache-airflow`
Then `./docs/start_doc_server.sh`
You will also see result of building the docs here in CI .
More info here: https://github.com/apache/airflow/blob/main/docs/README.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#discussion_r776812121
##########
File path: docs/apache-airflow/concepts/params.rst
##########
@@ -18,46 +18,115 @@
Params
======
-Params are Airflow's concept of providing runtime configuration to tasks when a DAG gets triggered manually.
-Params are configured while defining the DAG & tasks, that can be altered while doing a manual trigger. The
-ability to update params while triggering a DAG depends on the flag ``core.dag_run_conf_overrides_params``,
-so if that flag is ``False``, params would behave like constants.
+Params are Airflow's way to provide runtime configuration to tasks when a DAG gets triggered manually.
+To use them, initialize your DAG with a dictionary where the keys are strings with each param's name, and the values are ``Param`` objects.
-To use them, one can use the ``Param`` class for complex trigger-time validations or simply use primitive types,
-which won't be doing any such validations.
+``Param`` makes use of `json-schema <https://json-schema.org/>`, so one can use the full json-schema specifications mentioned at https://json-schema.org/draft/2020-12/json-schema-validation.html to define the construct of a ``Param`` objects.
+Or, if you want a default value without any validation, you can use literals instead.
.. code-block::
+ :caption a simple DAG with a parameter
Review comment:
```suggestion
:caption: a simple DAG with a parameter
```
Check: https://www.sphinx-doc.org/en/1.4.9/markup/code.html#caption-and-name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003117576
Static check failure looks unrelated, but the doc failure is related.
Do rebase on main and build docs again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#discussion_r776812199
##########
File path: docs/apache-airflow/concepts/params.rst
##########
@@ -18,46 +18,115 @@
Params
======
-Params are Airflow's concept of providing runtime configuration to tasks when a DAG gets triggered manually.
-Params are configured while defining the DAG & tasks, that can be altered while doing a manual trigger. The
-ability to update params while triggering a DAG depends on the flag ``core.dag_run_conf_overrides_params``,
-so if that flag is ``False``, params would behave like constants.
+Params are Airflow's way to provide runtime configuration to tasks when a DAG gets triggered manually.
+To use them, initialize your DAG with a dictionary where the keys are strings with each param's name, and the values are ``Param`` objects.
-To use them, one can use the ``Param`` class for complex trigger-time validations or simply use primitive types,
-which won't be doing any such validations.
+``Param`` makes use of `json-schema <https://json-schema.org/>`, so one can use the full json-schema specifications mentioned at https://json-schema.org/draft/2020-12/json-schema-validation.html to define the construct of a ``Param`` objects.
+Or, if you want a default value without any validation, you can use literals instead.
.. code-block::
+ :caption a simple DAG with a parameter
+ from airflow import DAG
+ from airflow.models.param import Param
+ from airflow.operators.python_operator import PythonOperator
+
+ with DAG(
+ "params",
+ params={"x": Param(5, type="integer", minimum=3),
+ "y": 6},
+ ) as the_dag:
+
+ def print_x(**context):
+ print(context["params"]["x"])
+
+ # prints 5, or whatever the user provided at trigger time
+ PythonOperator(
+ task_id="print_x",
+ python_callable=print_it,
+ )
+
+Params can also be added to individual tasks.
+If there's already a dag param with that name, the task-level default will take precedence over the dag-level default.
+
+.. code-block::
+ :caption tasks can have parameters too
+
+ # prints 10, or whatever the user provided at trigger time
+ PythonOperator(
+ task_id="print_x",
+ params={"x": 10},
+ python_callable=print_it,
+ )
+
+When a user manually triggers a dag, they can change the parameters that are provided to the dagrun.
+This can be disabled by setting ``core.dag_run_conf_overrides_params = False``, which will prevent the user from changing the params.
+If the user-supplied values don't pass validation, Airflow will show the user a warning instead of creating the dagrun.
+
+
+You can reference dag params via a templated task argument:
+
+.. code-block::
+ :caption use a template
+ from airflow import DAG
+ from airflow.models.param import Param
+ from airflow.operators.python import PythonOperator
+
+ with DAG(
+ "my_dag",
+ params={
+ # a int with a default value
+ "int_param": Param(10, type="integer", minimum=0, maximum=20),
+
+ # a required param which can be of multiple types
+ "dummy": Param(type=["null", "number", "string"]),
+
+ # a param which uses json-schema formatting
+ "email": Param(
+ default="example@example.com",
+ type="string",
+ format="idn-email",
+ minLength=5,
+ maxLength=255,
+ ),
+ },
+
+ # instead of getting strings from templates, get objects
+ render_template_as_native_obj=True,
+
+ ) as my_dag:
+
+ PythonOperator(
+ task_id="from_template",
+ op_args=[
+ "{{ params.int_param + 10 }}",
+ ],
+ python_callable=(
+ lambda x: print(type(x), x)
+ # '<class 'str'> 20' by default
+ # '<class 'int'> 20' if render_template_as_native_obj=True
+ ),
+ )
+
+By default, Jinja templates create strings.
+So if you have parameters that aren't strings, and you want to use templated task arguments, you might be interested in the ``render_template_as_native_obj`` DAG kwarg.
+It will allow you to preserve the type of the parameter, even if you manipulate it in a template.
+
+If templates aren't your style, you can access params in via the context.
+
+.. code-block::
+ :caption use the context kwarg
Review comment:
and here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService edited a comment on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService edited a comment on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003417611
Thanks for the tips @potiuk
When I do that (on Ubuntu 20.04, in a fresh venv) I get some errors:
```
#################### Running docs building ####################
apache-airflow : Building documentation
apache-airflow : Running sphinx. The output is hidden until an error occurs.
apache-airflow : Finished docs building with errors
#################### Output for documentation build apache-airflow ####################
apache-airflow : ################################################################################
apache-airflow [01mRunning Sphinx v3.4.3[39;49;00m
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 417, in load_extension
apache-airflow mod = import_module(extname)
apache-airflow File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
apache-airflow return _bootstrap._gcd_import(name, package, level)
apache-airflow File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
apache-airflow File "<frozen importlib._bootstrap>", line 991, in _find_and_load
apache-airflow File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
apache-airflow File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
apache-airflow File "<frozen importlib._bootstrap_external>", line 843, in exec_module
apache-airflow File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
apache-airflow File "/opt/airflow/docs/exts/exampleinclude.py", line 31, in <module>
apache-airflow from sphinx.ext.viewcode import viewcode_anchor
apache-airflow ImportError: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py)
apache-airflow
apache-airflow The above exception was the direct cause of the following exception:
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
apache-airflow app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 245, in __init__
apache-airflow self.setup_extension(extension)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 402, in setup_extension
apache-airflow self.registry.load_extension(self, extname)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 420, in load_extension
apache-airflow raise ExtensionError(__('Could not import extension %s') % extname,
apache-airflow sphinx.errors.ExtensionError: Could not import extension exampleinclude (exception: cannot import name
'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
apache-airflow
apache-airflow [91mExtension error:[39;49;00m
apache-airflow Could not import extension exampleinclude (exception: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode'
(/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
```
After which I can run the server, but I see a 404 error when I navigate here http://localhost:8000/docs/apache-airflow/latest/index.html
It seems to work OK on my Macbook though, so I'll just use that for now (though I'm happy to help toubleshoot the above error if you'd like, I'm just not sure where to start).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003421427
When I do that (on Ubuntu 20.04, in a fresh venv) I get some errors:
Did you run this?
```
./breeze build-docs -- --package-filter apache-airflow
```
If so - make sure to pull&build the image when asked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService edited a comment on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService edited a comment on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003417611
Thanks for the tips @potiuk
When I do that (on Ubuntu) I get some errors:
```
#################### Running docs building ####################
apache-airflow : Building documentation
apache-airflow : Running sphinx. The output is hidden until an error occurs.
apache-airflow : Finished docs building with errors
#################### Output for documentation build apache-airflow ####################
apache-airflow : ################################################################################
apache-airflow [01mRunning Sphinx v3.4.3[39;49;00m
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 417, in load_extension
apache-airflow mod = import_module(extname)
apache-airflow File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
apache-airflow return _bootstrap._gcd_import(name, package, level)
apache-airflow File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
apache-airflow File "<frozen importlib._bootstrap>", line 991, in _find_and_load
apache-airflow File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
apache-airflow File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
apache-airflow File "<frozen importlib._bootstrap_external>", line 843, in exec_module
apache-airflow File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
apache-airflow File "/opt/airflow/docs/exts/exampleinclude.py", line 31, in <module>
apache-airflow from sphinx.ext.viewcode import viewcode_anchor
apache-airflow ImportError: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py)
apache-airflow
apache-airflow The above exception was the direct cause of the following exception:
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
apache-airflow app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 245, in __init__
apache-airflow self.setup_extension(extension)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 402, in setup_extension
apache-airflow self.registry.load_extension(self, extname)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 420, in load_extension
apache-airflow raise ExtensionError(__('Could not import extension %s') % extname,
apache-airflow sphinx.errors.ExtensionError: Could not import extension exampleinclude (exception: cannot import name
'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
apache-airflow
apache-airflow [91mExtension error:[39;49;00m
apache-airflow Could not import extension exampleinclude (exception: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode'
(/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
```
After which I can run the server, but I see a 404 error when I navigate here http://localhost:8000/docs/apache-airflow/latest/index.html
It seems to work OK on my mac though, so I'll just use that for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003417611
Thanks for the tips @potiuk
When I do that I get some errors:
```#################### Running docs building ####################
apache-airflow : Building documentation
apache-airflow : Running sphinx. The output is hidden until an error occurs.
apache-airflow : Finished docs building with errors
#################### Output for documentation build apache-airflow ####################
apache-airflow : ################################################################################
apache-airflow [01mRunning Sphinx v3.4.3[39;49;00m
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 417, in load_extension
apache-airflow mod = import_module(extname)
apache-airflow File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
apache-airflow return _bootstrap._gcd_import(name, package, level)
apache-airflow File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
apache-airflow File "<frozen importlib._bootstrap>", line 991, in _find_and_load
apache-airflow File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
apache-airflow File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
apache-airflow File "<frozen importlib._bootstrap_external>", line 843, in exec_module
apache-airflow File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
apache-airflow File "/opt/airflow/docs/exts/exampleinclude.py", line 31, in <module>
apache-airflow from sphinx.ext.viewcode import viewcode_anchor
apache-airflow ImportError: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py)
apache-airflow
apache-airflow The above exception was the direct cause of the following exception:
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
apache-airflow app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 245, in __init__
apache-airflow self.setup_extension(extension)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 402, in setup_extension
apache-airflow self.registry.load_extension(self, extname)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 420, in load_extension
apache-airflow raise ExtensionError(__('Could not import extension %s') % extname,
apache-airflow sphinx.errors.ExtensionError: Could not import extension exampleinclude (exception: cannot import name
'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
apache-airflow
apache-airflow [91mExtension error:[39;49;00m
apache-airflow Could not import extension exampleinclude (exception: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode'
(/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
```
After which I can run the server, but I see a 404 error when I navigate here http://localhost:8000/docs/apache-airflow/latest/index.html
It seems to work OK on my mac though, so I'll just use that for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#discussion_r777798358
##########
File path: docs/apache-airflow/concepts/params.rst
##########
@@ -15,50 +15,140 @@
specific language governing permissions and limitations
under the License.
+.. _concepts:params:
+
Params
======
-Params are Airflow's concept of providing runtime configuration to tasks when a DAG gets triggered manually.
-Params are configured while defining the DAG & tasks, that can be altered while doing a manual trigger. The
-ability to update params while triggering a DAG depends on the flag ``core.dag_run_conf_overrides_params``,
-so if that flag is ``False``, params would behave like constants.
+Params are how Airflow provides runtime configuration to tasks.
+When you trigger a DAG manually, you can modify its Params before the dagrun starts.
+If the user-supplied values don't pass validation, Airflow shows a warning instead of creating the dagrun.
+(For scheduled runs, the default values are used.)
+
+Adding Params to a DAG
+----------------------
-To use them, one can use the ``Param`` class for complex trigger-time validations or simply use primitive types,
-which won't be doing any such validations.
+To add Params to a :class:`~airflow.models.dag.DAG`, initialize it with the ``params`` kwarg.
+Use a dictionary that maps Param names to a either a :class:`~airflow.models.param.Param` or an object indicating the parameter's default value.
.. code-block::
from airflow import DAG
from airflow.models.param import Param
with DAG(
- 'my_dag',
+ "the_dag",
+ params={"x": Param(5, type="integer", minimum=3),
+ "y": 6},
+ ) as the_dag:
Review comment:
```suggestion
"the_dag",
params={
"x": Param(5, type="integer", minimum=3),
"y": 6
},
) as the_dag:
```
nit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService edited a comment on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService edited a comment on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003417611
Thanks for the tips @potiuk
When I do that (on Ubuntu) I get some errors:
```
#################### Running docs building ####################
apache-airflow : Building documentation
apache-airflow : Running sphinx. The output is hidden until an error occurs.
apache-airflow : Finished docs building with errors
#################### Output for documentation build apache-airflow ####################
apache-airflow : ################################################################################
apache-airflow [01mRunning Sphinx v3.4.3[39;49;00m
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 417, in load_extension
apache-airflow mod = import_module(extname)
apache-airflow File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
apache-airflow return _bootstrap._gcd_import(name, package, level)
apache-airflow File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
apache-airflow File "<frozen importlib._bootstrap>", line 991, in _find_and_load
apache-airflow File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
apache-airflow File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
apache-airflow File "<frozen importlib._bootstrap_external>", line 843, in exec_module
apache-airflow File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
apache-airflow File "/opt/airflow/docs/exts/exampleinclude.py", line 31, in <module>
apache-airflow from sphinx.ext.viewcode import viewcode_anchor
apache-airflow ImportError: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py)
apache-airflow
apache-airflow The above exception was the direct cause of the following exception:
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
apache-airflow app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 245, in __init__
apache-airflow self.setup_extension(extension)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 402, in setup_extension
apache-airflow self.registry.load_extension(self, extname)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 420, in load_extension
apache-airflow raise ExtensionError(__('Could not import extension %s') % extname,
apache-airflow sphinx.errors.ExtensionError: Could not import extension exampleinclude (exception: cannot import name
'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
apache-airflow
apache-airflow [91mExtension error:[39;49;00m
apache-airflow Could not import extension exampleinclude (exception: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode'
(/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
```
After which I can run the server, but I see a 404 error when I navigate here http://localhost:8000/docs/apache-airflow/latest/index.html
It seems to work OK on my mac though, so I'll just use that for now (though I'm happy to help toubleshoot the above error if you'd like).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003441051
I don't know how I overlooked that prompt (several times). It's working now, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003111601
That looks good to me but you need to fix static checks and docs builds .
For static checks `pre-commit` is your friend :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService edited a comment on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService edited a comment on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003417611
Thanks for the tips @potiuk
When I do that (on Ubuntu) I get some errors:
```
#################### Running docs building ####################
apache-airflow : Building documentation
apache-airflow : Running sphinx. The output is hidden until an error occurs.
apache-airflow : Finished docs building with errors
#################### Output for documentation build apache-airflow ####################
apache-airflow : ################################################################################
apache-airflow [01mRunning Sphinx v3.4.3[39;49;00m
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 417, in load_extension
apache-airflow mod = import_module(extname)
apache-airflow File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
apache-airflow return _bootstrap._gcd_import(name, package, level)
apache-airflow File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
apache-airflow File "<frozen importlib._bootstrap>", line 991, in _find_and_load
apache-airflow File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
apache-airflow File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
apache-airflow File "<frozen importlib._bootstrap_external>", line 843, in exec_module
apache-airflow File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
apache-airflow File "/opt/airflow/docs/exts/exampleinclude.py", line 31, in <module>
apache-airflow from sphinx.ext.viewcode import viewcode_anchor
apache-airflow ImportError: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py)
apache-airflow
apache-airflow The above exception was the direct cause of the following exception:
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
apache-airflow app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 245, in __init__
apache-airflow self.setup_extension(extension)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 402, in setup_extension
apache-airflow self.registry.load_extension(self, extname)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 420, in load_extension
apache-airflow raise ExtensionError(__('Could not import extension %s') % extname,
apache-airflow sphinx.errors.ExtensionError: Could not import extension exampleinclude (exception: cannot import name
'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
apache-airflow
apache-airflow [91mExtension error:[39;49;00m
apache-airflow Could not import extension exampleinclude (exception: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode'
(/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
```
After which I can run the server, but I see a 404 error when I navigate here http://localhost:8000/docs/apache-airflow/latest/index.html
It seems to work OK on my Macbook though, so I'll just use that for now (though I'm happy to help toubleshoot the above error if you'd like, I'm just not sure where to start).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService edited a comment on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService edited a comment on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1003417611
Thanks for the tips @potiuk
When I do that (on Ubuntu) I get some errors:
```
#################### Running docs building ####################
apache-airflow : Building documentation
apache-airflow : Running sphinx. The output is hidden until an error occurs.
apache-airflow : Finished docs building with errors
#################### Output for documentation build apache-airflow ####################
apache-airflow : ################################################################################
apache-airflow [01mRunning Sphinx v3.4.3[39;49;00m
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 417, in load_extension
apache-airflow mod = import_module(extname)
apache-airflow File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
apache-airflow return _bootstrap._gcd_import(name, package, level)
apache-airflow File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
apache-airflow File "<frozen importlib._bootstrap>", line 991, in _find_and_load
apache-airflow File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
apache-airflow File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
apache-airflow File "<frozen importlib._bootstrap_external>", line 843, in exec_module
apache-airflow File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
apache-airflow File "/opt/airflow/docs/exts/exampleinclude.py", line 31, in <module>
apache-airflow from sphinx.ext.viewcode import viewcode_anchor
apache-airflow ImportError: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py)
apache-airflow
apache-airflow The above exception was the direct cause of the following exception:
apache-airflow
apache-airflow Traceback (most recent call last):
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
apache-airflow app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 245, in __init__
apache-airflow self.setup_extension(extension)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 402, in setup_extension
apache-airflow self.registry.load_extension(self, extname)
apache-airflow File "/usr/local/lib/python3.8/site-packages/sphinx/registry.py", line 420, in load_extension
apache-airflow raise ExtensionError(__('Could not import extension %s') % extname,
apache-airflow sphinx.errors.ExtensionError: Could not import extension exampleinclude (exception: cannot import name
'viewcode_anchor' from 'sphinx.ext.viewcode' (/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
apache-airflow
apache-airflow [91mExtension error:[39;49;00m
apache-airflow Could not import extension exampleinclude (exception: cannot import name 'viewcode_anchor' from 'sphinx.ext.viewcode'
(/usr/local/lib/python3.8/site-packages/sphinx/ext/viewcode.py))
```
After which I can run the server, but I see a 404 error when I navigate here http://localhost:8000/docs/apache-airflow/latest/index.html
It seems to work OK on my Macbook though, so I'll just use that for now (though I'm happy to help toubleshoot the above error if you'd like).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on pull request #20567: params concept doc rework
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on pull request #20567:
URL: https://github.com/apache/airflow/pull/20567#issuecomment-1004210636
I think I should've made this a draft PR until I was ready. Sorry about the extra noise.
Anyhow, I think that it is now actually ready for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org