[jira] [Created] (SPARK-34264) Prevent incomplete master URLs for Spark on Kubernetes early

["Jacek Laskowski (Jira)" <ji...@apache.org>]

Jacek Laskowski created SPARK-34264:
---------------------------------------

             Summary: Prevent incomplete master URLs for Spark on Kubernetes early
                 Key: SPARK-34264
                 URL: https://issues.apache.org/jira/browse/SPARK-34264
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes, Spark Submit
    Affects Versions: 3.0.1, 3.1.1
            Reporter: Jacek Laskowski


It turns out that {{--master k8s://}} is accepted and although leads to termination displays stacktraces that don't really tell what the real cause is.

This may happen when the Kubernetes API server(s) are described by an environment variable that's not initialized in the current terminal.

{code}
$ ./bin/spark-shell --master k8s:// --verbose
...
Spark config:
(spark.jars,)
(spark.app.name,Spark shell)
(spark.submit.pyFiles,)
(spark.ui.showConsoleProgress,true)
(spark.submit.deployMode,client)
(spark.master,k8s://https://)
...
21/01/27 14:29:44 ERROR Main: Failed to initialize Spark session.
io.fabric8.kubernetes.client.KubernetesClientException: Failed to start websocket
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:208)
...
Caused by: java.net.UnknownHostException: api: nodename nor servname provided, or not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
	at okhttp3.Dns$1.lookup(Dns.java:40)
	at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org