You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/02 23:36:09 UTC
[GitHub] [airflow] kaxil opened a new pull request #16786: Use Serialization to support more objects in XCom
kaxil opened a new pull request #16786:
URL: https://github.com/apache/airflow/pull/16786
closes https://github.com/apache/airflow/issues/8703
This way we will support Set, Tuple (and others like DAG,
BaseOperator, relativedelta, datetime) in XCom in serialization and
de-serialization.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #16786: Use Serialization to support more objects in XCom
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #16786:
URL: https://github.com/apache/airflow/pull/16786#issuecomment-873376186
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #16786: Use Serialization to support more objects in XCom
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #16786:
URL: https://github.com/apache/airflow/pull/16786#issuecomment-875486272
Maybe we need to start adding info against XCom rows, as I mentioned in https://github.com/apache/airflow/pull/16823#issuecomment-875407947 -- that way this could be used for new XComs but old ones would stay as the were.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #16786: Use Serialization to support more objects in XCom
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #16786:
URL: https://github.com/apache/airflow/pull/16786#issuecomment-903191311
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #16786: Use Serialization to support more objects in XCom
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #16786:
URL: https://github.com/apache/airflow/pull/16786#issuecomment-875485644
Yeah, this _toally_ breaks all existing JSON XComs:
```
In [1]: from airflow.serialization.serialized_objects import BaseSerialization
In [2]: BaseSerialization.from_dict({"x": "y"})
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-2-5a1376cdcc27> in <module>
----> 1 BaseSerialization.from_dict({"x": "y"})
~/code/airflow/airflow/airflow/serialization/serialized_objects.py in from_dict(cls, serialized_obj)
136 reconstructs all DAGs and operators it contains.
137 """
--> 138 return cls._deserialize(serialized_obj)
139
140 @classmethod
~/code/airflow/airflow/airflow/serialization/serialized_objects.py in _deserialize(cls, encoded_var)
269 if not isinstance(encoded_var, dict):
270 raise ValueError(f"The encoded_var should be dict and is {type(encoded_var)}")
--> 271 var = encoded_var[Encoding.VAR]
272 type_ = encoded_var[Encoding.TYPE]
273
KeyError: <Encoding.VAR: '__var'>
In [3]: BaseSerialization.from_dict({"x": "y"})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #16786: Use Serialization to support more objects in XCom
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #16786:
URL: https://github.com/apache/airflow/pull/16786#discussion_r665239836
##########
File path: airflow/models/xcom.py
##########
@@ -223,7 +223,9 @@ def serialize_value(value: Any):
if conf.getboolean('core', 'enable_xcom_pickling'):
return pickle.dumps(value)
try:
- return json.dumps(value).encode('UTF-8')
+ from airflow.serialization.serialized_objects import BaseSerialization
+
+ return json.dumps(BaseSerialization._serialize(value, fail_on_error=True)).encode('UTF-8')
Review comment:
It already exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #16786: Use Serialization to support more objects in XCom
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #16786:
URL: https://github.com/apache/airflow/pull/16786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #16786: Use Serialization to support more objects in XCom
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #16786:
URL: https://github.com/apache/airflow/pull/16786#discussion_r665240220
##########
File path: airflow/models/xcom.py
##########
@@ -223,7 +223,9 @@ def serialize_value(value: Any):
if conf.getboolean('core', 'enable_xcom_pickling'):
return pickle.dumps(value)
try:
- return json.dumps(value).encode('UTF-8')
+ from airflow.serialization.serialized_objects import BaseSerialization
+
+ return json.dumps(BaseSerialization._serialize(value, fail_on_error=True)).encode('UTF-8')
Review comment:
Oh, excetp:
```python
@classmethod
def to_json(cls, var: Union[DAG, BaseOperator, dict, list, set, tuple]) -> str:
"""Stringifies DAGs and operators contained by var and returns a JSON string of var."""
return json.dumps(cls.to_dict(var), ensure_ascii=True)
@classmethod
def to_dict(cls, var: Union[DAG, BaseOperator, dict, list, set, tuple]) -> dict:
"""Stringifies DAGs and operators contained by var and returns a dict of var."""
# Don't call on this class directly - only SerializedDAG or
# SerializedBaseOperator should be used as the "entrypoint"
raise NotImplementedError()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #16786: Use Serialization to support more objects in XCom
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #16786:
URL: https://github.com/apache/airflow/pull/16786#discussion_r663500550
##########
File path: airflow/models/xcom.py
##########
@@ -223,7 +223,9 @@ def serialize_value(value: Any):
if conf.getboolean('core', 'enable_xcom_pickling'):
return pickle.dumps(value)
try:
- return json.dumps(value).encode('UTF-8')
+ from airflow.serialization.serialized_objects import BaseSerialization
+
+ return json.dumps(BaseSerialization._serialize(value, fail_on_error=True)).encode('UTF-8')
Review comment:
Maybe we should add a `BaseSerialization.to_json()` method for symmetry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org