[GitHub] [airflow] dimon222 opened a new issue #16611: Pod name with period is causing issues for some apps in k8s

[GitBox <gi...@apache.org>]

dimon222 opened a new issue #16611:
URL: https://github.com/apache/airflow/issues/16611


   **Apache Airflow version**: 2.0.0+
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): All versions affected
   
   **Environment**:
   It affects all possible host configurations.
   What scripts will actually be affected inside of KubernetesPodOperator container - big question. In my scenario it was locally executed **Apache Spark**.
   
   **What happened**:
   
   This issue is consequence of change that was introduced in this commit/line:
   https://github.com/apache/airflow/commit/862443f6d3669411abfb83082c29c2fad7fcf12d#diff-01764c9ba7b2270764a59e7ff281c95809071b1f801170ee75a02481a8a730aaR475
   
   Pod operator is generating a pod name that has period in it. Whatever pod name is picked gets inherited by container itself, as result it becomes a hostname of it. The problem of hostnames in Linux is that if it contains period, it immediately becomes assumed a valid domain that DNS should be able to resolve. md5 digest in this weird case becomes assumed as first level "domain". Obviously, some libraries have no idea what to do with DNS domain like `airflow-pod-operator.9b702530e25f40f2b1cf6220842280c`, so they throw exceptions.
   In my use case the component was barking was **Apache Spark** in local mode.
   
   **What you expected to happen**:
   Apache Spark just works without issues and able to resolve itself by hostname without any code changes.
   
   **How to reproduce it**:
   As I'm not certain about full list of affected applications, I would for now assume anything that tries to resolve "current pod's" hostname.
   In my scenario I was running Wordcount of Apache Spark in local mode in KubernetesPodOperator. Perhaps, there might be easier ways to replicate it.
   
   **Anything else we need to know**:
   Having this kind of unresolvable real domain vs hostname confusion in my opinion is very very bad, and should be discouraged.
   
   The way for me to mitigate this issue right now was to build my own inheritance for KubernetesPodOperator that has method `create_pod_request_obj` overriden to call older method of generation of unique identifiers for pod name that used `-` (hyphen) instead of `.` (period) notation in name.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr closed issue #16611: Pod name with period is causing issues for some apps in k8s

[GitBox <gi...@apache.org>]

uranusjr closed issue #16611:
URL: https://github.com/apache/airflow/issues/16611


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #16611: Pod name with period is causing issues for some apps in k8s

[GitBox <gi...@apache.org>]

potiuk commented on issue #16611:
URL: https://github.com/apache/airflow/issues/16611#issuecomment-869582878


   Hey @dimon222 - would you be willing to add PR fixing it ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org