You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/15 09:36:59 UTC
[GitHub] [airflow] doowhtron opened a new issue #13692: task failed with a null hostname
doowhtron opened a new issue #13692:
URL: https://github.com/apache/airflow/issues/13692
**Apache Airflow version**: 2.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**: tencent cloud
- **OS** (e.g. from /etc/os-release): centos7
- **Kernel** (e.g. `uname -a`): 3.10
- **Install tools**:
- **Others**: Server version: 8.0.22 MySQL Community Server - GPL
**What happened**:
Task is in the **failed** state. I found the log file on one of the worker node and the task is actually success. And in the **task instance details** tab, the hostname field is null.
And the logs are as follows:
```
*** Log file does not exist: /data/app/epic-airflow/logs/tiny_demo80677608236/task_0/2021-01-15T09:04:00+00:00/1.log
*** Fetching from: http://:8793/log/tiny_demo80677608236/task_0/2021-01-15T09:04:00+00:00/1.log
*** Failed to fetch log file from worker. Invalid URL 'http://:8793/log/tiny_demo80677608236/task_0/2021-01-15T09:04:00+00:00/1.log': No host supplied
```
**What you expected to happen**:
Task should be success.
**How to reproduce it**:
I don't know how to reproduce it steadyly. It happened sometimes.
**Anything else we need to know**:
No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng edited a comment on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
huozhanfeng edited a comment on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-871773873
> Everyone facing the issue, please provide some more information about the setup of your Airflow installation, and if possible, steps to reproduce.
Just run a dag and let its task runs fail to reproduce the bug. More details in #16729. It's a bug of the scheduler, I will give a PR to solve it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] doowhtron commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
doowhtron commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-800085669
> Need more information about the setup of your Airflow Installation and if possible steps to reproduce
It still happens after I upgrade Airflow to 2.0.1
Maybe it is caused by the situation that a single task is scheduled more than once? I have four schedulers running on four hosts and two of them scheduled the same dag nearly the same time? Maybe a bug happened?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hafid-d commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
hafid-d commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-845058681
Getting the same issue as well on airflow 2.0.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sakirepallisrivamsi commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
sakirepallisrivamsi commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-860228936
Facing a similar issue on celery worker as well. Airflow 2.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng edited a comment on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
huozhanfeng edited a comment on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-871773873
> Everyone facing the issue, please provide some more information about the setup of your Airflow installation, and if possible, steps to reproduce.
Just run a dag and let its task runs fail to reproduce the bug. More details in #16729. It's a bug in the scheduler, I will give a PR to solve it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng edited a comment on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
huozhanfeng edited a comment on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-871773873
> Everyone facing the issue, please provide some more information about the setup of your Airflow installation, and if possible, steps to reproduce.
Just run a dag and let its task runs failed to reproduce the bug. More details in #16729. It's a bug of the scheduler, I have resolved it in my local env and will give a PR to solve it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-871421140
Everyone facing the issue, please provide some more information about the setup of your Airflow installation, and if possible, steps to reproduce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-1046195900
> happens on Airflow2.2.4rc1
Almost for sure it is an environmental/deployment issue. But I would love to the bottom of it so I will need your help @changxiaoju.
Does it happen all the time or only sporadically ? How often? Any details on the conditions that it happens ?
Can you check and possibly re-configure your hostname callable @changxiaoju https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable
Those issues can happen if your host cannot rertrieve hostname quickly enough - for example when your DNS experiences slow responses, delays, latency etc. Can you double check please if the hostname_callable of yours responds quickly (for example by repetitive calling it and making sure that you get the right response and report to use what are your findings?
Also stating what kind of deployment and configuration you have woudl be actually helpful. Specifically also how your DNS works, whether it is a cloud deployment etc?
You might want to use another method if your DNS is slow to respond and has "lagss"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] doowhtron commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
doowhtron commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-800089435
Is it possible to have a mechanism prevent a task from scheduling twice?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] changxiaoju commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
changxiaoju commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-1046768378
>
well, it worked several times and happened again,
> > happens on Airflow2.2.4rc1
>
> Almost for sure it is an environmental/deployment issue. But I would love to the bottom of it so I will need your help @changxiaoju.
>
> Does it happen all the time or only sporadically ? How often? Any details on the conditions that it happens ?
>
> Can you check and possibly re-configure your hostname callable @changxiaoju https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable
>
> Those issues can happen if your host cannot rertrieve hostname quickly enough - for example when your DNS experiences slow responses, delays, latency etc. Can you double check please if the hostname_callable of yours responds quickly (for example by repetitive calling it and making sure that you get the right response and report to use what are your findings?
>
> Also stating what kind of deployment and configuration you have woudl be actually helpful. Specifically also how your DNS works, whether it is a cloud deployment etc?
>
> You might want to use another method if your DNS is slow to respond and has "lagss"
Shut down for the same reson again, it is really hard to understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-1052721422
> I really want to offer you more information, but i am not very familiar with what you said. It happened all the time, i use it on remote server, the calculation clusters. And i only changed the postgresql related settings in airflow.cfg.
I think you need to run at the environmental stuff - we cannot help to solve those if we see no logs pointing to the issues. If you manage a software on k8s you should really be able to take a look at the logs of k8s - not only the logs of application - I think the reason is there - but it's a bit beyond the scope of Airlfow. You just need to review the logs of your environment to look for some anomalies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
huozhanfeng commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-871773873
> Everyone facing the issue, please provide some more information about the setup of your Airflow installation, and if possible, steps to reproduce.
Just run a dag and let its task runs fail to reproduce the bug. More details in #16729.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hafid-d edited a comment on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
hafid-d edited a comment on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-845058681
Getting the same issue as well on airflow 2.0.2. Anyone solved it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] doowhtron edited a comment on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
doowhtron edited a comment on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-800085669
> Need more information about the setup of your Airflow Installation and if possible steps to reproduce
It still happens after I upgrade Airflow to 2.0.1
Maybe it is caused by the situation that a single task is scheduled more than once? I have four schedulers running on four hosts and two of them scheduled the same dag nearly the same time? Maybe a bug happened?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-1013869060
Is anyone experiencing this issue on Airflow 2.2.3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-1040924209
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] emukans commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
emukans commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-804903679
I noticed the same issue on airflow 2.0.1 when I used cron notation for specifying schedule_interval. Here is my DAG:
```
DAG(
"id",
default_args=default_args,
description="description",
schedule_interval="0 14 * * *",
start_date=datetime(2021, 3, 23)
)
```
I wanted to specify the time. I solved the issue by using DateTimeSensor or TimeSensor and changing `schedule_interval="@daily"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-1013869060
Is anyone experiencing this issue on Airflow 2.2.3? If so please share reproduce steps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] changxiaoju commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
changxiaoju commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-1046167057
happens on Airflow2.2.4rc1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] changxiaoju commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
changxiaoju commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-1046263733
>
thankyou potiuk and https://github.com/apache/airflow/issues/18239 @[dimon222](https://github.com/dimon222).
it is now ok by setting hostname_callable = airflow.utils.net.get_host_ip_address.
I really want to offer you more information, but i am not very familiar with what you said.
It happened all the time, i use it on remote server, the calculation clusters. And i only changed the postgresql related settings in airflow.cfg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-1081269892
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-763201724
Need more information about the setup of your Airflow Installation and if possible steps to reproduce
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pparthesh commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
pparthesh commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-848777394
Getting the same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huozhanfeng edited a comment on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
huozhanfeng edited a comment on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-871773873
> Everyone facing the issue, please provide some more information about the setup of your Airflow installation, and if possible, steps to reproduce.
Just run a dag and let its task runs fail to reproduce the bug. More details in #16729. It's a bug of the scheduler, I have resolved it in my local env and will give a PR to solve it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] victorouttes commented on issue #13692: task failed with a null hostname
Posted by GitBox <gi...@apache.org>.
victorouttes commented on issue #13692:
URL: https://github.com/apache/airflow/issues/13692#issuecomment-857082866
Same issue here. Airflow 1.10.15 on kubernetes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org