You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ranga2crazyy (via GitHub)" <gi...@apache.org> on 2023/02/07 09:56:21 UTC
[GitHub] [airflow] ranga2crazyy opened a new issue, #29400: Network Error when changing DAG task instance status in UI
ranga2crazyy opened a new issue, #29400:
URL: https://github.com/apache/airflow/issues/29400
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
While changing the DAG's task instance(any) status as below, Airflow UI throws "Network error". But task instance's status does change though there is an error. It happens every time when one tries to change the TI status.
Not limited to Geo location.
Success to Failed OR Failed to Success
### What you think should happen instead
There should be no error pop up in UI
Enabled debug and analysed NGINX and gunicorn logs bit not useful. I could see HTTP post request in gunicorn logs to change the status as failed.
> 127.0.0.1 - - [07/Feb/2023:09:36:51 +0000] "POST /failed HTTP/1.1" 302 217 "<truncated>task_id=load" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) App leWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
> 127.0.0.1 - - [07/Feb/2023:09:36:52 +0000] "POST /failed HTTP/1.1" 302 217 "<truncated>&task_id=load" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
debug logs from nginx error log file
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http upstream request: "/failed?"
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http upstream process header
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 malloc: 0000561549221F00:4096
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 recv: eof:0, avail:1
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 recv: fd:18 617 of 4096
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 **http proxy status 302 "302 FOUND"**
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy header: "Server: gunicorn"
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy header: "Date: Tue, 07 Feb 2023 09:36:52 GMT"
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy header: "Connection: close"
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy header: "Content-Type: text/html; charset=utf-8"
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy header: "Content-Length: 217"
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy header: "Location: <Airflow_DNS>/home"
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy header: "X-Robots-Tag: noindex, nofollow"
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy header: "Set-Cookie: session=9ccd9591-1c77-4751-8f3a-9742d7965b1c.FaWqilm3H_PHBYT6s4Zbms42M-Q; Expires=Tue, 07-Feb-2023 10:06:52 GMT; HttpOnly; Path=/; SameSite=Lax"
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy header done
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 HTTP/1.1 302 FOUND
> Server: nginx/1.14.1
> Date: Tue, 07 Feb 2023 09:36:52 GMT
> Content-Type: text/html; charset=utf-8
> Content-Length: 217
> Connection: keep-alive
> Location: <Airflow_DNS>/home
> X-Robots-Tag: noindex, nofollow
> Set-Cookie: session=9ccd9591-1c77-4751-8f3a-9742d7965b1c.FaWqilm3H_PHBYT6s4Zbms42M-Q; Expires=Tue, 07-Feb-2023 10:06:52 GMT; HttpOnly; Path=/; SameSite=Lax
>
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 write new buf t:1 f:0 000056154925AC08, pos 000056154925AC08, size: 409 file: 0, size: 0
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http write filter: l:0 f:0 s:409
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http cacheable: 0
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http proxy filter init s:302 h:0 c:0 l:217
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 http upstream process upstream
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 pipe read upstream: 0
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 pipe preread: 217
> 2023/02/07 09:36:52 [debug] 1162293#0: *84162 pipe buf free s:0 t:1 f:0 0000561549221F00, pos 0000561549222090, size: 217 file: 0, size: 0
### How to reproduce
Env details provided in deployment section. this error has not been observed in older airflow version which we used(2.2.1 & 3.6).
1. Login to UI.
2. Nav to any DAG.
3. try to change any task instance's status. Error will pop up down bottom
4. Click on confirm and again error reappears.
5. Cancel and refresh the page, status would have changed as intended.
### Operating System
RHEL 8.7 (Ootpa)
### Versions of Apache Airflow Providers
Providers info
apache-airflow-providers-ftp | 2.1.2
apache-airflow-providers-http | 2.1.2
apache-airflow-providers-imap | 2.2.3
apache-airflow-providers-microsoft-mssql | 2.1.1
apache-airflow-providers-postgres | 3.0.0
apache-airflow-providers-snowflake | 2.5.1
apache-airflow-providers-sqlite | 2.1.3
### Deployment
Virtualenv installation
### Deployment details
Apache Airflow
**version | 2.3.2**
executor | CeleryExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn | postgresql
AWS EC2 System info
OS | Linux
architecture | x86_64
python_version | 3.9.13 (main, Nov 9 2022, 13:16:24) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)]
python_location | /usr/bin/python3.9
nginx version: nginx/1.14.1
gunicorn==20.1.0
gunicorn worker class = sync
NGINX conf
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name <DNS> ;
ssl_certificate <truncated>
ssl_certificate_key <truncated>
location / {
proxy_pass http://localhost:8080;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
### Anything else
Every time it occurs.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29400: Network Error when changing DAG task instance status in UI
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29400:
URL: https://github.com/apache/airflow/issues/29400#issuecomment-1420494732
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ranga2crazyy commented on issue #29400: Network Error when changing DAG task instance status in UI
Posted by "ranga2crazyy (via GitHub)" <gi...@apache.org>.
ranga2crazyy commented on issue #29400:
URL: https://github.com/apache/airflow/issues/29400#issuecomment-1451632932
@ephraimbuddy I am not able to reproduce it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ranga2crazyy commented on issue #29400: Network Error when changing DAG task instance status in UI
Posted by "ranga2crazyy (via GitHub)" <gi...@apache.org>.
ranga2crazyy commented on issue #29400:
URL: https://github.com/apache/airflow/issues/29400#issuecomment-1429804102
I have not faced this in previous versions prior to 2.3.2. Ever since we upgraded to 2.3.2, this is happening on all our envs. It is not an occasional error.
I will test this on latest main and let you know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #29400: Network Error when changing DAG task instance status in UI
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #29400:
URL: https://github.com/apache/airflow/issues/29400#issuecomment-1427678529
Can't reproduce in 2.5.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29400: Network Error when changing DAG task instance status in UI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29400:
URL: https://github.com/apache/airflow/issues/29400#issuecomment-1454900170
Closing it then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ranga2crazyy commented on issue #29400: Network Error when changing DAG task instance status in UI
Posted by "ranga2crazyy (via GitHub)" <gi...@apache.org>.
ranga2crazyy commented on issue #29400:
URL: https://github.com/apache/airflow/issues/29400#issuecomment-1420497078
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29400: Network Error when changing DAG task instance status in UI
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29400: Network Error when changing DAG task instance status in UI
URL: https://github.com/apache/airflow/issues/29400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org