You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/19 15:30:36 UTC
[GitHub] [airflow] awildturtok opened a new issue #17084: Resources cannot be used due to issues with pickling
awildturtok opened a new issue #17084:
URL: https://github.com/apache/airflow/issues/17084
**Apache Airflow version**:
2.0.0
**Environment**:
- **OS** (e.g. from /etc/os-release): Container using Ubuntu Groovy 20.10
- **Kernel** (e.g. `uname -a`): 4.18.0-305.7.1.el8_4.x86_64
**What happened**:
When using resources in a dag, it has some issues related to resources and pickling.
When I add the following to any working Task:
```
resources={"ram" : 15000}
```
it breaks with the following message
```
File "/usr/local/lib/python3.8/dist-packages/airflow/serialization/serialized_objects.py", line 184, in serialize_to_json
if cls._is_excluded(value, key, object_to_serialize):
File "/usr/local/lib/python3.8/dist-packages/airflow/serialization/serialized_objects.py", line 501, in _is_excluded
return super()._is_excluded(var, attrname, op)
File "/usr/local/lib/python3.8/dist-packages/airflow/serialization/serialized_objects.py", line 170, in _is_excluded
return isinstance(var, cls._excluded_types) or cls._value_is_hardcoded_default(
File "/usr/local/lib/python3.8/dist-packages/airflow/serialization/serialized_objects.py", line 328, in _value_is_hardcoded_default
cls._CONSTRUCTOR_PARAMS[attrname] is value or (value in [{}, []])
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/operator_resources.py", line 136, in __eq__
return self.__dict__ == other.__dict__
AttributeError: 'dict' object has no attribute '__dict__'
```
**What you expected to happen**:
Using resources shouldn't be an issue. As far as I tracked the error down, it looks like _is_excluded makes wrong assumptions about the self object causing problems.
**How to reproduce it**:
```
def do_echo():
logging.info("hi")
with DAG(
'an_echoing_dag',
schedule_interval=None,
max_active_runs=1,
start_date=datetime(2020, 1, 1),
) as dag:
PythonOperator(
task_id="task",
python_callable=do_echo,
resources={"ram" : 15000}
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] awildturtok commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
awildturtok commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-883134545
I think I can do that. Where would be a sensible place to add de-/serialization tests for resources? Or a test that checks exactly this code path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-1003092789
@awildturtok Would you still want to work on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-882643963
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] awildturtok commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
awildturtok commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-883134545
I think I can do that. Where would be a sensible place to add de-/serialization tests for resources? Or a test that checks exactly this code path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-883212852
Sounds reasonable to add a serialisation test case for the currently failing scenario (and then implement the fix to make the scenario to not raise an exception).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-883044707
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-882643963
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-883044707
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-882643963
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] awildturtok commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
awildturtok commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-883134545
I think I can do that. Where would be a sensible place to add de-/serialization tests for resources? Or a test that checks exactly this code path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #17084: Resources cannot be used due to issues with pickling
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #17084:
URL: https://github.com/apache/airflow/issues/17084#issuecomment-883044707
I think this can basically be fixed by changing `Resources.__eq__()` to something like
```python
if not isintance(other, type(self)):
return False
return self.__dict__ == other.__dict__
```
The same should probably also be done for `Resource.__eq__()`. Would you be ionterested in submitting a PR for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org