You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/07 18:54:40 UTC
[GitHub] [airflow] lewismc opened a new pull request #20755: Augment xcom docs
lewismc opened a new pull request #20755:
URL: https://github.com/apache/airflow/pull/20755
This PR addresses some [investigation and subsequent assistance on user@](https://lists.apache.org/thread/qpfkk2ktdm8gn78t3gvoqkxwoh218s98) regarding the development and deployment of custom XCom backends.
This is only a documentation patch.
Thanks for any review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1009396549
Sure. The code starting this line needs to be valid python code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#discussion_r797079825
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,63 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
+
+ from airflow.settings import conf
+ conf.get("core", "xcom_backend")
+
+If using env vars check with ``env|grep AIRFLOW__CORE__XCOM``.
Review comment:
(this didn't end up being applied)
Do we even need this? The `conf.get` will pull it from the env var already, right?
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,50 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+.. code-block:: python
+
+ from airflow.models.xcom import XCom
+
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
+
+.. code-block:: python
+
+ from airflow.settings import conf
+
+ conf.get("core", "xcom_backend")
+
+If using env vars check with ``env|grep AIRFLOW__CORE__XCOM``.
+
+Working with Custom Backends in K8s via Helm
+--------------------------------------------
+
+Running custom XCom backends in K8s will introduce even more complexity to you Airflow deployment. Put simply, sometimes things go wrong which can be difficult to debug.
+
+For example, if you define a custom XCom backend in the Chart ``values.yaml`` (via the ``xcom_backend`` configuration) and Airflow fails to load the class, the entire Chart deployment will fail with each pod container attempting to restart time and time again.
+
+When deploying in K8s your custom XCom backend needs to be reside in a ``config`` directory otherwise it cannot be located during Chart deployment.
+
+An observed problem is that it is very difficult to acquire logs from the container because there is a very small window of availability where the trace can be obtained. The only way you can determine the root cause is if you are fortunate enough to query and acquire the container logs at the right time. This in turn prevents the entire Helm chart from deploying successfully.
Review comment:
```suggestion
An observed problem is that it is very difficult to acquire logs from the container because there is a very small window of availability where the trace can be obtained. The only way you can determine the root cause is if you are fortunate enough to query and acquire the container logs at the right time. This in turn prevents the entire Helm chart from deploying successfully.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1028541526
Thanks again @jedcunningham
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on a change in pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on a change in pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#discussion_r780629491
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,63 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
+
+ from airflow.settings import conf
+ conf.get("core", "xcom_backend")
+
+If using env vars check with ``env|grep AIRFLOW__CORE__XCOM``.
Review comment:
Thanks for the peer review TP. It's been a while since I used rst. This looks much better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1023683095
@potiuk finally :) thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#discussion_r798106661
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,50 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+.. code-block:: python
+
+ from airflow.models.xcom import XCom
+
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
Review comment:
```suggestion
You can also examine Airflow's configuration:
```
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,50 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+.. code-block:: python
+
+ from airflow.models.xcom import XCom
+
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
+
+.. code-block:: python
+
+ from airflow.settings import conf
+
+ conf.get("core", "xcom_backend")
+
+If using env vars, check with ``env|grep AIRFLOW__CORE__XCOM``.
Review comment:
```suggestion
```
Since `conf.get` will also show the value from the env var, I don't think we need this.
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,50 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
Review comment:
```suggestion
Firstly, if you can exec into a terminal in the container then you should be able to do:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1007882424
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1009375859
@potiuk thanks for the nudge :)
I see the following
> blacken-docs.............................................................................Failed
> - hook id: blacken-docs
> - exit code: 1
> docs/apache-airflow/concepts/xcoms.rst:91: code block parse error Cannot parse: 1:16: Traceback (most recent call last):
However I have no idea what to fix in line 91 of `xcoms.rst`. Any ideas? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on a change in pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on a change in pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#discussion_r781669385
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,69 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+.. code-block:: python
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
+
+.. code-block:: python
+
+ from airflow.settings import conf
+ conf.get("core", "xcom_backend")
+
+If using env vars check with ``env|grep AIRFLOW__CORE__XCOM``.
+
+Working with Custom Backends in K8s via Helm
+--------------------------------------------
+
+Running custom XCom backends in K8s will introduce even more complexity to you Airflow deployment. Put simply, sometimes things go wrong which can be difficult to debug.
+
+For example, if you define a custom XCom backend in the Chart ``values.yaml`` (via the ``xcom_backend`` configuration) and Airflow fails to load the class, the entire Chart deployment will fail with each pod container attempting to restart time and time again.
+
+When deploying in K8s your custom XCom backend needs to be reside in a ``config`` directory otherwise it cannot be located during Chart deployment.
+
+An observed problem is that it is very difficult to acquire logs from the container because there is a very small window of availability where the trace can be obtained. If you are fortunate enough to query the container logs at the right time, assuming that the custom backend value used is ``xcom_custom_backend.S3XComBackend``, you may see something similar to the following::
+
+.. code-block:: python
+
+ Traceback (most recent call last):
Review comment:
How silly of me! Thanks for pointing that out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1019388836
Hi @potiuk I made some changes lets see if CI likes them or not. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1019369457
Would you rebase and fix static/docs errors?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#discussion_r781612835
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,69 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+.. code-block:: python
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
+
+.. code-block:: python
+
+ from airflow.settings import conf
+ conf.get("core", "xcom_backend")
+
+If using env vars check with ``env|grep AIRFLOW__CORE__XCOM``.
+
+Working with Custom Backends in K8s via Helm
+--------------------------------------------
+
+Running custom XCom backends in K8s will introduce even more complexity to you Airflow deployment. Put simply, sometimes things go wrong which can be difficult to debug.
+
+For example, if you define a custom XCom backend in the Chart ``values.yaml`` (via the ``xcom_backend`` configuration) and Airflow fails to load the class, the entire Chart deployment will fail with each pod container attempting to restart time and time again.
+
+When deploying in K8s your custom XCom backend needs to be reside in a ``config`` directory otherwise it cannot be located during Chart deployment.
+
+An observed problem is that it is very difficult to acquire logs from the container because there is a very small window of availability where the trace can be obtained. If you are fortunate enough to query the container logs at the right time, assuming that the custom backend value used is ``xcom_custom_backend.S3XComBackend``, you may see something similar to the following::
+
+.. code-block:: python
+
+ Traceback (most recent call last):
Review comment:
This is not valid python code. Don't mark it as python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1028176706
Thanks for the review @jedcunningham
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1007882343
@uranusjr I also didn't know I could merge your suggestion. I've never used that GH feature before. That's pretty cool. Thanks for taking the time to make the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on a change in pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on a change in pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#discussion_r780629454
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,63 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
Review comment:
Thanks for the peer review TP. It's been a while since I used rst. This looks much better.
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,63 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
+
+ from airflow.settings import conf
+ conf.get("core", "xcom_backend")
+
+If using env vars check with ``env|grep AIRFLOW__CORE__XCOM``.
Review comment:
Thanks for the peer review TP. It's been a while since I used rst. This looks much better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on a change in pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on a change in pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#discussion_r780629454
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,63 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
Review comment:
Thanks for the peer review TP. It's been a while since I used rst. This looks much better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1007882343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1028471825
Noticed a few more things, overall LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1009035929
some build docs and static checks are failing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1019598663
Still some errors, I am afraid. I recommend to install `pre-commit` locally and run it for static checks and then run docs build (for example using `./breeze build-docs` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham merged pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
jedcunningham merged pull request #20755:
URL: https://github.com/apache/airflow/pull/20755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#discussion_r780623923
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,63 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
Review comment:
```suggestion
Firstly, if you can exec into a terminal in the container then you should be able to do:
.. code-block:: python
from airflow.models.xcom import XCom
print(XCom.__name__)
```
Better syntax highlighting. Same for the conf code blocks below.
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,63 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
+
+ from airflow.settings import conf
+ conf.get("core", "xcom_backend")
+
+If using env vars check with ``env|grep AIRFLOW__CORE__XCOM``.
Review comment:
```suggestion
If using env vars, check with ``env|grep AIRFLOW__CORE__XCOM``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1007882424
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lewismc commented on pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#issuecomment-1007882389
... and the "nitpickings" are what make good codebases much better. Thanks for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #20755: Augment xcom docs
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20755:
URL: https://github.com/apache/airflow/pull/20755#discussion_r780623923
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,63 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
Review comment:
```suggestion
Firstly, if you can exec into a terminal in the container then you should be able to do:
.. code-block:: python
from airflow.models.xcom import XCom
print(XCom.__name__)
```
Better syntax highlighting. Same for the conf code blocks below.
##########
File path: docs/apache-airflow/concepts/xcoms.rst
##########
@@ -42,13 +42,63 @@ XComs are a relative of :doc:`variables`, with the main difference being that XC
Note: If the first task run is not succeeded then on every retry task XComs will be cleared to make the task run idempotent.
-Custom Backends
----------------
+Custom XCom Backends
+--------------------
The XCom system has interchangeable backends, and you can set which backend is being used via the ``xcom_backend`` configuration option.
If you want to implement your own backend, you should subclass :class:`~airflow.models.xcom.BaseXCom`, and override the ``serialize_value`` and ``deserialize_value`` methods.
There is also an ``orm_deserialize_value`` method that is called whenever the XCom objects are rendered for UI or reporting purposes; if you have large or expensive-to-retrieve values in your XComs, you should override this method to avoid calling that code (and instead return a lighter, incomplete representation) so the UI remains responsive.
-You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend process the data lifecycle easier.
+You can also override the ``clear`` method and use it when clearing results for given dags and tasks. This allows the custom XCom backend to process the data lifecycle easier.
+
+Working with Custom XCom Backends in Containers
+-----------------------------------------------
+
+Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can be useful to be assured that a custom XCom backend is actually being initialized. For example, the complexity of the container environment can make it more difficult to determine if your backend is being loaded correctly during container deployment. Luckily the following guidance can be used to assist you in building confidence in your custom XCom implementation.
+
+Firstly, if you can exec into a terminal in the container then you should be able to do::
+
+ from airflow.models.xcom import XCom
+ print(XCom.__name__)
+
+which will print the actual class that is being used.
+
+Depending on how you've configured the backend, you can also examine airflow
+configuration::
+
+ from airflow.settings import conf
+ conf.get("core", "xcom_backend")
+
+If using env vars check with ``env|grep AIRFLOW__CORE__XCOM``.
Review comment:
```suggestion
If using env vars, check with ``env|grep AIRFLOW__CORE__XCOM``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org