You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/06/28 08:33:00 UTC

[jira] [Commented] (AIRFLOW-4862) Allow directly using IP address as hostname in airflow.utils.net.get_hostname()

    [ https://issues.apache.org/jira/browse/AIRFLOW-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874774#comment-16874774 ] 

ASF GitHub Bot commented on AIRFLOW-4862:
-----------------------------------------

XD-DENG commented on pull request #5501: [AIRFLOW-4862] Allow directly using IP address as hostname
URL: https://github.com/apache/airflow/pull/5501
 
 
   ### Jira
   
     - https://issues.apache.org/jira/browse/AIRFLOW-4862
   
   
   ### Description
   
   There are some cases in which components need to talk to each other (e.g., webserver fetches log from worker), but the hostname from `airflow.utils.net.get_hostname()` may not always be resolvable.
   
   This PR provide the choice to let use use IP address directly as hostname, so there will be no DNS resolving issue.
   
   For mode details, like the real issue I encountered in my environments, please refer to description in the JIRA ticket https://issues.apache.org/jira/browse/AIRFLOW-4862
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Allow directly using IP address as hostname in airflow.utils.net.get_hostname()
> -------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-4862
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4862
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: utils
>    Affects Versions: 1.10.3
>            Reporter: Xiaodong DENG
>            Assignee: Xiaodong DENG
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> In airflow.utils.net.get_hostname(), the default function used to get hostname for nodes (like worker) is *socket.getfqdn()*, which will return fully qualified domain name.
> In some cases, we do need to ensure that hostnames can resolved so that nodes can talk to each other. One example is: if I use S3 for remote logging, then the log will only be pushed to S3 after the job is finished (either success or failure); when the job is still running, webserver will first check if the log is available in its own volume, if not, webserver will fetch log from worker.
> If workers' hostnames are something like "airflow-worker-53-4bp8v" (e.g., when running on OpenShift or K8S), it's possible that the hostname can't be resolved, then we will observe errors like below
> {code:java}
> *** Log file does not exist: /opt/app-root/airflow/logs/example_python_operator/sleep_for_3/2019-06-18T08:14:15.313472+00:00/2.log
> *** Fetching from: http://airflow-worker-57-n69vb:8793/log/example_python_operator/sleep_for_3/2019-06-18T08:14:15.313472+00:00/2.log
> *** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-57-n69vb', port=8793): Max retries exceeded with url: /log/example_python_operator/sleep_for_3/2019-06-18T08:14:15.313472+00:00/2.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fec3266a6d8>: Failed to establish a new connection: [Errno -2] Name or service not known',))
> {code}
>  
> This may be addressed by properly setting service discovery, but the users may not always have the privilege to do that (like myself in my organization).
> Another solution is to change "hostname_callable" in Airflow's configuration ([core] section). But it will only work if you have a function which can return a resolvable hostname and it can't take any argument (due to the existing implementation [https://github.com/apache/airflow/blob/dd08ae3469a50a145f9ae7f819ed1840fe2a5bd6/airflow/utils/net.py#L41-L45).]
>  
> The change I would like to propose is: allow users to use IP address directly as hostname.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)