You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/09 04:04:49 UTC
[GitHub] [airflow] Bowrna opened a new pull request, #28251: task docker documentation added
Bowrna opened a new pull request, #28251:
URL: https://github.com/apache/airflow/pull/28251
closes: #25926
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #28251:
URL: https://github.com/apache/airflow/pull/28251#discussion_r1046776893
##########
docs/apache-airflow-providers-docker/decorators/docker.rst:
##########
@@ -0,0 +1,130 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/decorator:docker:
+
+Task Docker Decorator
+=====================
+
+Python callable wrapped within the ``@task.docker`` decorator with args are executed within
+the docker container.
+
+Parameters
+----------
+
+The following parameters are supported in Docker Task decorator.
Review Comment:
This list of parameters is long and duplicated from API documentation. It may be difficult to sync the two sources. Would it be possible to use autodoc to automatically pull this content from docstrings in Python files? Or maybe it’d be better to say the decorator takes all the same parameters as DockerOperator (except `task_id`), and link to the DockerOperator’s API documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna closed pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
Bowrna closed pull request #28251: task docker documentation added
URL: https://github.com/apache/airflow/pull/28251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tiokim commented on a diff in pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
tiokim commented on code in PR #28251:
URL: https://github.com/apache/airflow/pull/28251#discussion_r1045320242
##########
docs/apache-airflow-providers-docker/decorators/docker.rst:
##########
@@ -0,0 +1,130 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/decorator:docker:
+
+Task Docker Decorator
+=====================
+
+Python callable wrapped within the ``@task.docker`` decorator with args are executed within
+the docker container.
+
+Parameters
+----------
+
+The following parameters are supported in Docker Task decorator.
+
+multiple_outputs
+ If set, function return value will be unrolled to multiple XCom values.
+ Dict will unroll to XCom values with keys as XCom keys. Defaults to False.
+use_dill
+ Whether to use dill or pickle for serialization
+python_command
+ Python command for executing functions, Default python3
+image
+ Docker image from which to create the container.
+ If image tag is omitted, "latest" will be used.
+api_version
+ Remote API version. Set to ``auto`` to automatically detect the server's version.
+container_name
+ Name of the container. Optional (templated)
+cpus:
+ Number of CPUs to assign to the container. This value gets multiplied with 1024.
+docker_url
+ URL of the host running the docker daemon.
+ Default is unix://var/run/docker.sock
+environment
+ Environment variables to set in the container. (templated)
+private_environment
+ Private environment variables to set in the container.
+ These are not templated, and hidden from the website.
+force_pull
+ Pull the docker image on every run. Default is False.
+mem_limit
+ Maximum amount of memory the container can use.
+ Either a float value, which represents the limit in bytes,
+ or a string like ``128m`` or ``1g``.
+host_tmp_dir
+ Specify the location of the temporary directory on the host which will
+ be mapped to tmp_dir. If not provided defaults to using the standard system temp directory.
+network_mode
+ Network mode for the container.
+
+ It can be one of the following:
+ bridge
+ Create new network stack for the container with default docker bridge network
+ 'None'
+ No networking for this container
+ container:<name> or <id>
+ Use the network stack of another container specified via <name> or <id>
+ host
+ Use the host network stack. Incompatible with `port_bindings`
+ '<network-name>' or '<network-id>'
+ Connects the container to user created network(using `docker network create` command)
+tls_ca_cert
+ Path to a PEM-encoded certificate authority to secure the docker connection.
+tls_client_cert
+ Path to the PEM-encoded certificate used to authenticate docker client.
+tls_client_key
+ Path to the PEM-encoded key used to authenticate docker client.
+tls_hostname
+ Hostname to match against the docker server certificate or False to disable the check.
+tls_ssl_version
+ Version of SSL to use when communicating with docker daemon.
+tmp_dir
+ Mount point inside the container to
+ a temporary directory created on the host by the operator.
+ The path is also made available via the environment variable
+ ``AIRFLOW_TMP_DIR`` inside the container.
+user
+ Default user inside the docker container.
+mounts
+ List of mounts to mount into the container, e.g.
+ ``['/host/path:/container/path', '/host/path2:/container/path2:ro']``.
+working_dir
+ Working directory to set on the container (equivalent to the -w switch the docker client)
+xcom_all
+ Push all the stdout or just the last line. The default is False (last line).
+docker_conn_id
+ ID of the Airflow connection to use
+dns
+ Docker custom DNS servers
+dns_search
+ Docker custom DNS search domain
+auto_remove
+ Auto-removal of the container on daemon side when the container's process exits.
+ The default is False.
Review Comment:
IMHO, [The default](https://github.com/apache/airflow/blob/b52ce7f483d02fe17467ca3fa13cfffdb92ac103/airflow/providers/docker/operators/docker.py#L134) seems `never`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
Bowrna commented on PR #28251:
URL: https://github.com/apache/airflow/pull/28251#issuecomment-1347892270
Is there any other changes that I have to do to get this PR merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a diff in pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
Bowrna commented on code in PR #28251:
URL: https://github.com/apache/airflow/pull/28251#discussion_r1046821560
##########
docs/apache-airflow-providers-docker/decorators/docker.rst:
##########
@@ -0,0 +1,130 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/decorator:docker:
+
+Task Docker Decorator
+=====================
+
+Python callable wrapped within the ``@task.docker`` decorator with args are executed within
+the docker container.
+
+Parameters
+----------
+
+The following parameters are supported in Docker Task decorator.
Review Comment:
yes i agree @uranusjr I tried using autodoc and it didn't work. I will try again and check or I will point to the DockerOperator's API documentation and mention about excluding task_id point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #28251:
URL: https://github.com/apache/airflow/pull/28251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28251:
URL: https://github.com/apache/airflow/pull/28251#issuecomment-1349055350
> After building the docs, how do I check in the browser if the built docs are rightly rendering in UI? Is there any documentation on that part?
https://github.com/apache/airflow/blob/main/docs/README.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a diff in pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
Bowrna commented on code in PR #28251:
URL: https://github.com/apache/airflow/pull/28251#discussion_r1054370415
##########
docs/apache-airflow-providers-docker/decorators/docker.rst:
##########
@@ -0,0 +1,130 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/decorator:docker:
+
+Task Docker Decorator
+=====================
+
+Python callable wrapped within the ``@task.docker`` decorator with args are executed within
+the docker container.
+
+Parameters
+----------
+
+The following parameters are supported in Docker Task decorator.
Review Comment:
i will see if i can fix this and raise as separate PR @uranusjr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
Bowrna commented on PR #28251:
URL: https://github.com/apache/airflow/pull/28251#issuecomment-1348790219
After building the docs, how do I check in the browser if the built docs are rightly rendering in UI? Is there any documentation on that part?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28251:
URL: https://github.com/apache/airflow/pull/28251#issuecomment-1344317061
Some static checks fixes needed though :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a diff in pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
Bowrna commented on code in PR #28251:
URL: https://github.com/apache/airflow/pull/28251#discussion_r1046770749
##########
docs/apache-airflow-providers-docker/decorators/docker.rst:
##########
@@ -0,0 +1,130 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/decorator:docker:
+
+Task Docker Decorator
+=====================
+
+Python callable wrapped within the ``@task.docker`` decorator with args are executed within
+the docker container.
+
+Parameters
+----------
+
+The following parameters are supported in Docker Task decorator.
+
+multiple_outputs
+ If set, function return value will be unrolled to multiple XCom values.
+ Dict will unroll to XCom values with keys as XCom keys. Defaults to False.
+use_dill
+ Whether to use dill or pickle for serialization
+python_command
+ Python command for executing functions, Default python3
+image
+ Docker image from which to create the container.
+ If image tag is omitted, "latest" will be used.
+api_version
+ Remote API version. Set to ``auto`` to automatically detect the server's version.
+container_name
+ Name of the container. Optional (templated)
+cpus:
+ Number of CPUs to assign to the container. This value gets multiplied with 1024.
+docker_url
+ URL of the host running the docker daemon.
+ Default is unix://var/run/docker.sock
+environment
+ Environment variables to set in the container. (templated)
+private_environment
+ Private environment variables to set in the container.
+ These are not templated, and hidden from the website.
+force_pull
+ Pull the docker image on every run. Default is False.
+mem_limit
+ Maximum amount of memory the container can use.
+ Either a float value, which represents the limit in bytes,
+ or a string like ``128m`` or ``1g``.
+host_tmp_dir
+ Specify the location of the temporary directory on the host which will
+ be mapped to tmp_dir. If not provided defaults to using the standard system temp directory.
+network_mode
+ Network mode for the container.
+
+ It can be one of the following:
+ bridge
+ Create new network stack for the container with default docker bridge network
+ 'None'
+ No networking for this container
+ container:<name> or <id>
+ Use the network stack of another container specified via <name> or <id>
+ host
+ Use the host network stack. Incompatible with `port_bindings`
+ '<network-name>' or '<network-id>'
+ Connects the container to user created network(using `docker network create` command)
+tls_ca_cert
+ Path to a PEM-encoded certificate authority to secure the docker connection.
+tls_client_cert
+ Path to the PEM-encoded certificate used to authenticate docker client.
+tls_client_key
+ Path to the PEM-encoded key used to authenticate docker client.
+tls_hostname
+ Hostname to match against the docker server certificate or False to disable the check.
+tls_ssl_version
+ Version of SSL to use when communicating with docker daemon.
+tmp_dir
+ Mount point inside the container to
+ a temporary directory created on the host by the operator.
+ The path is also made available via the environment variable
+ ``AIRFLOW_TMP_DIR`` inside the container.
+user
+ Default user inside the docker container.
+mounts
+ List of mounts to mount into the container, e.g.
+ ``['/host/path:/container/path', '/host/path2:/container/path2:ro']``.
+working_dir
+ Working directory to set on the container (equivalent to the -w switch the docker client)
+xcom_all
+ Push all the stdout or just the last line. The default is False (last line).
+docker_conn_id
+ ID of the Airflow connection to use
+dns
+ Docker custom DNS servers
+dns_search
+ Docker custom DNS search domain
+auto_remove
+ Auto-removal of the container on daemon side when the container's process exits.
+ The default is False.
Review Comment:
I verified and the default value is False @tiokim
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
Bowrna commented on PR #28251:
URL: https://github.com/apache/airflow/pull/28251#issuecomment-1345492223
I have fixed the static check issues. @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a diff in pull request #28251: task docker documentation added
Posted by GitBox <gi...@apache.org>.
Bowrna commented on code in PR #28251:
URL: https://github.com/apache/airflow/pull/28251#discussion_r1045650135
##########
docs/apache-airflow-providers-docker/decorators/docker.rst:
##########
@@ -0,0 +1,130 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/decorator:docker:
+
+Task Docker Decorator
+=====================
+
+Python callable wrapped within the ``@task.docker`` decorator with args are executed within
+the docker container.
+
+Parameters
+----------
+
+The following parameters are supported in Docker Task decorator.
+
+multiple_outputs
+ If set, function return value will be unrolled to multiple XCom values.
+ Dict will unroll to XCom values with keys as XCom keys. Defaults to False.
+use_dill
+ Whether to use dill or pickle for serialization
+python_command
+ Python command for executing functions, Default python3
+image
+ Docker image from which to create the container.
+ If image tag is omitted, "latest" will be used.
+api_version
+ Remote API version. Set to ``auto`` to automatically detect the server's version.
+container_name
+ Name of the container. Optional (templated)
+cpus:
+ Number of CPUs to assign to the container. This value gets multiplied with 1024.
+docker_url
+ URL of the host running the docker daemon.
+ Default is unix://var/run/docker.sock
+environment
+ Environment variables to set in the container. (templated)
+private_environment
+ Private environment variables to set in the container.
+ These are not templated, and hidden from the website.
+force_pull
+ Pull the docker image on every run. Default is False.
+mem_limit
+ Maximum amount of memory the container can use.
+ Either a float value, which represents the limit in bytes,
+ or a string like ``128m`` or ``1g``.
+host_tmp_dir
+ Specify the location of the temporary directory on the host which will
+ be mapped to tmp_dir. If not provided defaults to using the standard system temp directory.
+network_mode
+ Network mode for the container.
+
+ It can be one of the following:
+ bridge
+ Create new network stack for the container with default docker bridge network
+ 'None'
+ No networking for this container
+ container:<name> or <id>
+ Use the network stack of another container specified via <name> or <id>
+ host
+ Use the host network stack. Incompatible with `port_bindings`
+ '<network-name>' or '<network-id>'
+ Connects the container to user created network(using `docker network create` command)
+tls_ca_cert
+ Path to a PEM-encoded certificate authority to secure the docker connection.
+tls_client_cert
+ Path to the PEM-encoded certificate used to authenticate docker client.
+tls_client_key
+ Path to the PEM-encoded key used to authenticate docker client.
+tls_hostname
+ Hostname to match against the docker server certificate or False to disable the check.
+tls_ssl_version
+ Version of SSL to use when communicating with docker daemon.
+tmp_dir
+ Mount point inside the container to
+ a temporary directory created on the host by the operator.
+ The path is also made available via the environment variable
+ ``AIRFLOW_TMP_DIR`` inside the container.
+user
+ Default user inside the docker container.
+mounts
+ List of mounts to mount into the container, e.g.
+ ``['/host/path:/container/path', '/host/path2:/container/path2:ro']``.
+working_dir
+ Working directory to set on the container (equivalent to the -w switch the docker client)
+xcom_all
+ Push all the stdout or just the last line. The default is False (last line).
+docker_conn_id
+ ID of the Airflow connection to use
+dns
+ Docker custom DNS servers
+dns_search
+ Docker custom DNS search domain
+auto_remove
+ Auto-removal of the container on daemon side when the container's process exits.
+ The default is False.
Review Comment:
I will check this and update. @tiokim
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org