You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/20 00:18:10 UTC
[GitHub] [airflow] ajbosco opened a new issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
ajbosco opened a new issue #14327:
URL: https://github.com/apache/airflow/issues/14327
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 2.0.1
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): v1.17.12
**Environment**:
- **Cloud provider or hardware configuration**: AWS EKS
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
When you click on Graph view for some DAGs in 2.0.1 the UI errors out (logs below).
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
The Graph view to display
<!-- What do you think went wrong? -->
**How to reproduce it**:
It is not clear to me. This only happening on a handle of our DAGs. Also the tree view displays fine.
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
```
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 97, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 2080, in graph
return self.render_template(
File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 396, in render_template
return super().render_template(
File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/baseviews.py", line 280, in render_template
return render_template(
File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 137, in render_template
return _render(
File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/airflow/www/templates/airflow/graph.html", line 21, in top-level template code
{% from 'appbuilder/loading_dots.html' import loading_dots %}
File "/usr/local/lib/python3.8/site-packages/airflow/www/templates/airflow/dag.html", line 21, in top-level template code
{% from 'appbuilder/dag_docs.html' import dag_docs %}
File "/usr/local/lib/python3.8/site-packages/airflow/www/templates/airflow/main.html", line 20, in top-level template code
{% extends 'appbuilder/baselayout.html' %}
File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
{% import 'appbuilder/baselib.html' as baselib %}
File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 60, in top-level template code
{% block tail %}
File "/usr/local/lib/python3.8/site-packages/airflow/www/templates/airflow/graph.html", line 145, in block "tail"
var task_instances = {{ task_instances|tojson }};
File "/usr/local/lib/python3.8/site-packages/flask/json/__init__.py", line 376, in tojson_filter
return Markup(htmlsafe_dumps(obj, **kwargs))
File "/usr/local/lib/python3.8/site-packages/flask/json/__init__.py", line 290, in htmlsafe_dumps
dumps(obj, **kwargs)
File "/usr/local/lib/python3.8/site-packages/flask/json/__init__.py", line 211, in dumps
rv = _json.dumps(obj, **kwargs)
File "/usr/local/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.8/site-packages/airflow/utils/json.py", line 74, in _default
raise TypeError(f"Object of type '{obj.__class__.__name__}' is not JSON serializable")
TypeError: Object of type 'V1ResourceRequirements' is not JSON serializable
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mokubyow edited a comment on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
Mokubyow edited a comment on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-808497582
Just wanted to chime in here since I had the same exact issue. I found the issue was only present when using the deprecated executor_config format. When I upgraded to using the pod_override with the k8s models my graph view was able to render. So it appears to be due to something in the backward compatibility process. Example:
**Broken...**
```
PythonOperator(
task_id=f"sync_{table_name}",
python_callable=sync_table,
provide_context=True,
op_kwargs={"table_name": table_name},
executor_config={"KubernetesExecutor": {"request_cpu": "1"}},
retries=5,
dag=dag,
)
```
**Works!**
```
PythonOperator(
task_id=f"sync_{table_name}",
python_callable=sync_table,
provide_context=True,
op_kwargs={"table_name": table_name},
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(requests={"cpu": "1"}),
)
]
)
)
},
retries=5,
dag=dag,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] NilsJPWerner commented on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
NilsJPWerner commented on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-785341584
This same issue is happening for me. I only have python operators running on kuberetes executor and any time I try to look at the graph view the website breaks with the `Object of type 'V1ResourceRequirements' is not JSON serializable` error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] NilsJPWerner edited a comment on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
NilsJPWerner edited a comment on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-785342026
Here is the full traceback:
```
Ooops!
Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.
Python version: 3.7.10
Airflow version: 2.0.0+astro.3
Node:
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.7/site-packages/airflow/www/auth.py", line 34, in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/decorators.py", line 97, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/decorators.py", line 60, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2101, in graph
dag_run_state=dt_nr_dr_data['dr_state'],
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 400, in render_template
**kwargs,
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line 281, in render_template
template, **dict(list(kwargs.items()) + list(self.extra_args.items()))
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template
ctx.app,
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/graph.html", line 21, in top-level template code
{% from 'appbuilder/loading_dots.html' import loading_dots %}
File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/dag.html", line 21, in top-level template code
{% from 'appbuilder/dag_docs.html' import dag_docs %}
File "/usr/local/lib/python3.7/site-packages/astronomer/airflow/version_check/templates/astro-baselayout.html", line 1, in top-level template code
{% extends airflow_base_template %}
File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/main.html", line 20, in top-level template code
{% extends 'appbuilder/baselayout.html' %}
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
{% import 'appbuilder/baselib.html' as baselib %}
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 60, in top-level template code
{% block tail %}
File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/graph.html", line 145, in block "tail"
var task_instances = {{ task_instances|tojson }};
File "/usr/local/lib/python3.7/site-packages/flask/json/__init__.py", line 376, in tojson_filter
return Markup(htmlsafe_dumps(obj, **kwargs))
File "/usr/local/lib/python3.7/site-packages/flask/json/__init__.py", line 290, in htmlsafe_dumps
dumps(obj, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask/json/__init__.py", line 211, in dumps
rv = _json.dumps(obj, **kwargs)
File "/usr/local/lib/python3.7/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/json.py", line 74, in _default
raise TypeError(f"Object of type '{obj.__class__.__name__}' is not JSON serializable")
TypeError: Object of type 'V1ResourceRequirements' is not JSON serializable
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-783484826
@ajbosco Could you provide an example DAG that this fails on?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-805967652
@kaxil Would you be able to take a look at this? I don't think the root cause is in the UI.
It blows up in `graph.html` because thats the only spot we try to convert `task_instances` to json, but the `V1ResourceRequirements` isn't valid json so it all blows up. I imagine it would be best to address that before we get to the html templates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-806215897
`task_instances` is built from `alchemy_to_dict()`, which does not promise to return a JSON-serialisable object. So some additional processing is needed. (I don’t think it’s a good idea to make the change directly in `alchemy_to_dict()` either since the function is used for things not required to be JSON-serialisable.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-806215897
`task_instances` is built from `alchemy_to_dict()`, which does not promise to return a JSON-serialisable object. So some additional processing is needed. (I don’t think it’s a good idea to make the change directly in `alchemy_to_dict()` either since the function is used for things not required to be JSON-serialisation.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Mokubyow commented on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
Mokubyow commented on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-808497582
Just wanted to chime in here since I had the same exact issue. I found the issue was only present when using the deprecated executor_config format. When I upgraded to using the pod_override with the k8s models my graph view was able to render. So it appears to be due to something in the backward compatibility process. Example:
**Broken...**
```
PythonOperator(
task_id=f"sync_{table_name}",
python_callable=sync_table,
provide_context=True,
op_kwargs={"table_name": table_name},
executor_config={"KubernetesExecutor": {"request_cpu": "1"}},
retries=5,
dag=dag,
)
```
**Works!**
```
PythonOperator(
task_id=f"sync_{table_name}",
python_callable=sync_table,
provide_context=True,
op_kwargs={"table_name": table_name},
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(requests={"cpu": sync_cpu}),
)
]
)
)
},
retries=5,
dag=dag,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] NilsJPWerner commented on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
NilsJPWerner commented on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-785342026
Here is the full traceback:
```
Ooops!
Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.
Python version: 3.7.10
Airflow version: 2.0.0+astro.3
Node: elementary-mass-2343-webserver-77c8dcdcd9-jsszn
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.7/site-packages/airflow/www/auth.py", line 34, in decorated
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/decorators.py", line 97, in view_func
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/decorators.py", line 60, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2101, in graph
dag_run_state=dt_nr_dr_data['dr_state'],
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 400, in render_template
**kwargs,
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line 281, in render_template
template, **dict(list(kwargs.items()) + list(self.extra_args.items()))
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template
ctx.app,
File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/graph.html", line 21, in top-level template code
{% from 'appbuilder/loading_dots.html' import loading_dots %}
File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/dag.html", line 21, in top-level template code
{% from 'appbuilder/dag_docs.html' import dag_docs %}
File "/usr/local/lib/python3.7/site-packages/astronomer/airflow/version_check/templates/astro-baselayout.html", line 1, in top-level template code
{% extends airflow_base_template %}
File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/main.html", line 20, in top-level template code
{% extends 'appbuilder/baselayout.html' %}
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
{% import 'appbuilder/baselib.html' as baselib %}
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 60, in top-level template code
{% block tail %}
File "/usr/local/lib/python3.7/site-packages/airflow/www/templates/airflow/graph.html", line 145, in block "tail"
var task_instances = {{ task_instances|tojson }};
File "/usr/local/lib/python3.7/site-packages/flask/json/__init__.py", line 376, in tojson_filter
return Markup(htmlsafe_dumps(obj, **kwargs))
File "/usr/local/lib/python3.7/site-packages/flask/json/__init__.py", line 290, in htmlsafe_dumps
dumps(obj, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask/json/__init__.py", line 211, in dumps
rv = _json.dumps(obj, **kwargs)
File "/usr/local/lib/python3.7/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/json.py", line 74, in _default
raise TypeError(f"Object of type '{obj.__class__.__name__}' is not JSON serializable")
TypeError: Object of type 'V1ResourceRequirements' is not JSON serializable
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #14327:
URL: https://github.com/apache/airflow/issues/14327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ajbosco commented on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
ajbosco commented on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-784238745
It seems to occur on any DAG with a `KubernetesPodOperator` task with `resources` set. And only after a DAG Run exists.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thedrex commented on issue #14327: Kubernetes Objects are not serializable and break Graph View in UI
Posted by GitBox <gi...@apache.org>.
thedrex commented on issue #14327:
URL: https://github.com/apache/airflow/issues/14327#issuecomment-793938659
@bbovenzi - I'm experiencing this issue in 2.0.1 as well, so I've attached a sample DAG that will reproduce and generate the tracebacks copy/pasted above in this thread.
To reproduce: run a DAG with an operator task that passes resource requests to the KubernetesExecutor.
ie: `executor_config={"KubernetesExecutor": {"request_memory": "200Mi", "limit_memory": "500Mi"}}`
the graph view for the DAG will be available until the first dag run completes, then an internal server error occurs and further attempts to load the graph view get the traceback re: `TypeError: Object of type 'V1ResourceRequirements' is not JSON serializable`.
```
"""
graphbreaker_python_operator_dag is a dummy dag to illustrate an airflow 2.0.1 bug.
currently, operator tasks passing configs to the KubernetesExecutor break the graph
view of the UI.
see below for a PythonOperator example.
note: the graph view won't break until there's a least one existing dag run.
see: https://github.com/apache/airflow/issues/14327
"""
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import timedelta,datetime
dag_params = {
"owner": "airflow",
"depends_on_past": False,
"email": "airflow@example.com",
"email_on_failure": False,
"email_on_retry": False,
"retries": 1,
"start_date": datetime(2021, 3, 5),
"retry_delay": timedelta(minutes=5),
}
with DAG("graphbreaker_python_operator_dag", default_args=dag_params, schedule_interval=None, catchup=False) as dag:
def print_paths():
import pprint, sys
pprint.pp(sys.path)
break_graph_view = PythonOperator(
task_id="print_paths_and_break_graph",
executor_config={"KubernetesExecutor": {"request_memory": "200Mi", "limit_memory": "500Mi"}},
python_callable=print_paths,
dag=dag
)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org