You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2023/05/18 09:11:10 UTC
[GitHub] [airflow] amoghrajesh commented on a diff in pull request #31376: Issue-21638: Spark Connection with k8s in URL not mapped correctly
amoghrajesh commented on code in PR #31376:
URL: https://github.com/apache/airflow/pull/31376#discussion_r1197589507
##########
airflow/models/connection.py:
##########
@@ -186,16 +186,46 @@ def _normalize_conn_type(conn_type):
conn_type = conn_type.replace("-", "_")
return conn_type
+ @staticmethod
+ def split_uri_to_parts(uri):
+ c = uri.count("://")
+ if c < 2:
+ return urlsplit(uri), False
+ idx = uri.rfind("://")
+ # replace :// with :__
+ uri_list = list(uri)
+ uri_list[idx : idx + 3] = ":__"
+ uri = "".join(uri_list)
+ uri = urlsplit(uri)
+ uri = [e for e in uri]
+ # replace back :__ with ://
+ uri[1] = uri[1].replace(":__", "://")
+ uri = SplitResult(uri[0], uri[1], uri[2], uri[3], uri[4])
Review Comment:
Thanks for your comment. No, one line 200 we are doing a list comprehension. This is leading to the type being as `list`.
Verified by adding a few log statements:
```
def split_uri_to_parts(uri):
c = uri.count("://")
if c < 2:
return urlsplit(uri), False
idx = uri.rfind("://")
# replace :// with :__
uri_list = list(uri)
uri_list[idx : idx + 3] = ":__"
uri = "".join(uri_list)
uri = urlsplit(uri)
uri = [e for e in uri]
# replace back :__ with ://
uri[1] = uri[1].replace(":__", "://")
print(type(uri))
uri = SplitResult(uri[0], uri[1], uri[2], uri[3], uri[4])
print(type(uri))
return uri, True
```
Output is:
```
<class 'list'>
<class 'urllib.parse.SplitResult'>
(SplitResult(scheme='spark', netloc='k8s://100.68.0.1:443', path='', query='deploy-mode=cluster', fragment=''), True)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org