You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by suryag10 <gi...@git.apache.org> on 2018/09/16 04:38:19 UTC
[GitHub] spark pull request #22433: Support STS to run in k8s cluster mode
GitHub user suryag10 opened a pull request:
https://github.com/apache/spark/pull/22433
Support STS to run in k8s cluster mode
## What changes were proposed in this pull request?
Code is enhanced to allow the STS run in kubernetes deployment with spark deploy mode of cluster.
## How was this patch tested?
Started the sts in cluster mode in K8S deployment and was able to run some queries using the beeline client.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/suryag10/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22433.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22433
----
commit 3a7fa571181e4b0494f2b705fbd07bc61b0ca6ce
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-16T04:37:26Z
Support STS to run in k8s cluster mode
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22433
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96105/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
I am observing some weird behaviour when i am trying to respond to the comments. Hence i am adding the resposes to comments as below.
Following are the responses for the comments:
>>The script may be run from a client machine outside a k8s cluster. In this case, there's not even a pod. I would suggest separating the explanation of the user flow details by the deploy mode (client vs cluster).
STS is a server and its best way of deployment in K8S cluster is either done through the helm chart or through the yaml file(although it can be done through the method you had suggested, but i guess that scenario would be a rare case and there will be no HA of the STS server if it is triggered from outside).
>> In the scenario of a cluster-mode submission, what is the command-line behavior? Does the thrift-server script "block" until the thrift server pod is shut down?
By default the script returns but can be made to block by setting the environment variable SPARK_NO_DAEMONIZE. Once this is done, script blocks until the thrift server pod is shut down
>>If possible, there should be some basic integration testing. Run a thrift server command against the minishift cluster used by the other testing.
Will add it as a separate PR.
Pls merge this, if you are ok with the responses.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> Does it fail in k8s or does spark k8s code error out ?
> If former, why not fix “name” handling in k8s to replace unsupported characters ?
Following is the error seen without the fix:
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://k8s-apiserver.bcmt.cluster.local:8443/api/v1/namespaces/default/pods. Message: Pod "thrift jdbc/odbc server-1537079590890-driver" is invalid: metadata.name: Invalid value: "thrift jdbc/odbc server-1537079590890-driver": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'). Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.name, message=Invalid value: "thrift jdbc/odbc server-1537079590890-driver": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-
9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Pod, name=thrift jdbc/odbc server-1537079590890-driver, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "thrift jdbc/odbc server-1537079590890-driver" is invalid: metadata.name: Invalid value: "thrift jdbc/odbc server-1537079590890-driver": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
This is not specific to Kubernetes, but more of a generic DNS (DNS-1123)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r218943584
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Spark deploy mode of Client
--- End diff --
Can we change the section title to just `Client Deployment Mode`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by nrchakradhar <gi...@git.apache.org>.
Github user nrchakradhar commented on the issue:
https://github.com/apache/spark/pull/22433
This PR is now same as [PR-20272](https://github.com/apache/spark/pull/20272).
The conversation in [PR-20272](https://github.com/apache/spark/pull/20272), has some useful information which can be included in the Spark documentation
Also, its good to mention that STS will not work with dynamicAllocation as shuffle support is not yet available.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22433
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> In the scenario of a cluster-mode submission, what is the command-line behavior? Does the thrift-server script "block" until the thrift server pod is shut down?
By default the script returns but can be made to block by setting the environment variable SPARK_NO_DAEMONIZE. Once this is done, script blocks until the thrift server pod is shut down
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
@mridulm @liyinan926 @jacobdr @ifilonenko
code check for space,"/" handling is already present at https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L259
I had reverted back the fix in start-thriftserver.sh. Please review and merge.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by jacobdr <gi...@git.apache.org>.
Github user jacobdr commented on the issue:
https://github.com/apache/spark/pull/22433
> a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.'
Your changes to the name handling don’t comply with this, so agree with @mridulm you should move this change elsewhere and more broadly support name validation/sanitization for submitted applications in kubernetes
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
@liyinan926
>>The script may be run from a client machine outside a k8s cluster. In this case, there's not even a pod. >>I would suggest separating the explanation of the user flow details by the deploy mode (client vs cluster).
STS is a server and its best way of deployment in K8S cluster is either done through the helm chart or through the yaml file(although it can be done through the method you had suggested, but i guess that scenario would be a rare case and there will be no HA of the STS server if it is triggered from outside).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22433
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r218944068
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Spark deploy mode of Client
+
+To start STS in client mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
--- End diff --
Can we add put this into a code block?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r226898234
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,43 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Client Deployment Mode
+
+To start STS in client mode, excute the following command
+
+```bash
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>
+```
+
+### Cluster Deployment Mode
+
+To start STS in cluster mode, excute the following command
+
+```bash
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+ --deploy-mode cluster
+```
+
+The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name(pod/container from
--- End diff --
STS is a server and its best way of deployment in K8S cluster is either done through the helm chart or through the yaml file(although it can be done through the method you had suggested, but i guess that scenario would be a rare case and there will be no HA of the STS server if it is triggered from outside).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
Can some body pls merge this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> Thank you for your first contribution, @suryag10 .
>
> * Could you file a SPARK JIRA issue since this is a code change?
Sure.
> * Could you update the PR title like the other PRs? e.g. `[SPARK-XXX][SQL][K8S] ...`?
Sure.
>
> And, just out of curious, do we need this change?
>
> ```shell
> - exec "${SPARK_HOME}"/sbin/spark-daemon.sh submit $CLASS 1 --name "Thrift JDBC/ODBC Server" "$@"
> + exec "${SPARK_HOME}"/sbin/spark-daemon.sh submit $CLASS 1 --name "Thrift-JDBC-ODBC-Server" "$@"
> ```
Without the above change, it fails to start the driver pod as well. Spaces, "/" are not allowed for the "name" in the kubernetes world.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22433
**[Test build #96105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96105/testReport)** for PR 22433 at commit [`3a7fa57`](https://github.com/apache/spark/commit/3a7fa571181e4b0494f2b705fbd07bc61b0ca6ce).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r227067136
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,43 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Client Deployment Mode
+
+To start STS in client mode, excute the following command
+
+```bash
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>
+```
+
+### Cluster Deployment Mode
+
+To start STS in cluster mode, excute the following command
+
+```bash
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+ --deploy-mode cluster
+```
+
+The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name(pod/container from
--- End diff --
Yes, it might be rare to use this type of deployment on k8s, but it's still possible, e.g., for testing, to run the STS in client mode on a client machine. The wording here assumes that the server is running in a pod, which might not be the case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22433
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3135/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Posted by ifilonenko <gi...@git.apache.org>.
Github user ifilonenko commented on the issue:
https://github.com/apache/spark/pull/22433
test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22433
**[Test build #96105 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96105/testReport)** for PR 22433 at commit [`3a7fa57`](https://github.com/apache/spark/commit/3a7fa571181e4b0494f2b705fbd07bc61b0ca6ce).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22433
> As this script is common start point for all the resource managers(k8s/yarn/mesos/standalone/local), i guess changing this to fit for all the cases has a value add, instead of doing at each resource manager level. Thoughts?
Please note that I am specifically referring only to the need for changing application `name`.
The rationale given that `name` should be DNS compliant is a restriction specific to k8s and not spark.
Instead of doing one off rename's the right approach would be to handle this name translation such that it will benefit not just STS, but any user application.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r218944092
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Spark deploy mode of Client
+
+To start STS in client mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>
+
+### Spark deploy mode of Cluster
+
+To start STS in cluster mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
--- End diff --
Ditto.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22433
Thank you for your first contribution, @suryag10 .
- Could you file a SPARK JIRA issue since this is a code change?
- Could you update the PR title like the other PRs? e.g. `[SPARK-XXX][SQL][K8S] ...`?
And, just out of curious, do we need this change?
```bash
- exec "${SPARK_HOME}"/sbin/spark-daemon.sh submit $CLASS 1 --name "Thrift JDBC/ODBC Server" "$@"
+ exec "${SPARK_HOME}"/sbin/spark-daemon.sh submit $CLASS 1 --name "Thrift-JDBC-ODBC-Server" "$@"
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22433
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3135/
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r218943664
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Spark deploy mode of Client
+
+To start STS in client mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>
+
+### Spark deploy mode of Cluster
--- End diff --
Ditto, can we change this to `Cluster Deployment Mode`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
@liyinan926
>>The script may be run from a client machine outside a k8s cluster. In this case, there's not even a pod. >>I would suggest separating the explanation of the user flow details by the deploy mode (client vs cluster).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r219728772
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,43 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Client Deployment Mode
+
+To start STS in client mode, excute the following command
+
+```bash
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>
+```
+
+### Cluster Deployment Mode
+
+To start STS in cluster mode, excute the following command
+
+```bash
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+ --deploy-mode cluster
+```
+
+The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name(pod/container from
--- End diff --
The script may be run from a client machine outside a k8s cluster. In this case, there's not even a pod. I would suggest separating the explanation of the user flow details by the deploy mode (client vs cluster).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> > Agreed with @mridulm that the naming restriction is specific to k8s and should be handled in a k8s specific way, e.g., somewhere around https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L208.
>
> Ok, Will update the PR with the same.
Hi, Handling of this conversion is already present in
https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L259
I had reverted back the change in start-thriftserver.sh file. Please review and merge.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22433
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3135/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by erikerlandson <gi...@git.apache.org>.
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/22433
@suryag10, all things being equal, it is considered preferable to provide testing for new functionality on the same PR. Are there are logistical problems adding testing here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by suryag10 <gi...@git.apache.org>.
GitHub user suryag10 reopened a pull request:
https://github.com/apache/spark/pull/22433
[SPARK-25442][SQL][K8S] Support STS to run in k8s deployments with spark deployment mode as cluster.
## What changes were proposed in this pull request?
Code is enhanced to allow the STS run in kubernetes deployment with spark deploy mode of cluster.
## How was this patch tested?
Started the sts in cluster mode in K8S deployment and was able to run some queries using the beeline client.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/suryag10/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22433.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22433
----
commit 3a7fa571181e4b0494f2b705fbd07bc61b0ca6ce
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-16T04:37:26Z
Support STS to run in k8s cluster mode
commit 3556a61241e3f4910673f1bbf701905870ed09ea
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-16T04:37:26Z
[SPARK-25442][SQL][K8S] Support STS to run in k8s deployments with spark deployment mode as cluster.
commit 78dc1a35f299d854b61c9b03e22730960c6280a2
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-19T05:16:48Z
Merge branch 'master' of https://github.com/suryag10/spark
commit a15f5313e7c798e58c80147e575218bb70fe2d74
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-19T05:16:48Z
Merge branch 'master' of https://github.com/suryag10/spark
commit 42dd479a33279b85fb9b3fa5a70570970e8148a1
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-19T17:08:58Z
Merge branch 'master' of https://github.com/suryag10/spark
commit 4a7e737ae1210451de668b13a72bbd9473721f45
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-19T17:08:58Z
Merge branch 'master' of https://github.com/suryag10/spark
commit d91fa2badf33cc4122d340af64ef669dddc66cf1
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-19T17:17:20Z
Merge branch 'master' of https://github.com/suryag10/spark
commit a65cfa56a64a7b9fb78a038ce6f9d25f2ce0e428
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-19T17:17:20Z
Merge branch 'master' of https://github.com/suryag10/spark
commit 8dc7ced8e44c1d75a27a15a9605d2bd8a693a732
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-19T17:23:03Z
Merge branch 'master' of https://github.com/suryag10/spark
commit 6e021e7706c12103ec9ce08b20d7fcb66c83aeb2
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-19T17:23:03Z
Merge branch 'master' of https://github.com/suryag10/spark
commit 12be1d2364e852b7cc27b78c0aec9740693e5cab
Author: Suryanarayana GARLAPATI <su...@...>
Date: 2018-09-21T12:33:12Z
Merge branch 'master' of https://github.com/suryag10/spark
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> In the scenario of a cluster-mode submission, what is the command-line behavior? Does the thrift-server script "block" until the thrift server pod is shut down?
By default the script returns but can be made to block by setting the environment variable SPARK_NO_DAEMONIZE. Once this is done, script blocks until the thrift server pod is shut down
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
can somebody pls review and merge?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s cluster mode
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22433
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployments with spark deploym...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22433
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22433
Does it fail in k8s or does spark k8s code error out ?
If former, why not fix “name” handling in k8s to replace unsupported characters ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by erikerlandson <gi...@git.apache.org>.
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/22433
@suryag10 you were probably encountering github server problems from yesterday:
https://status.github.com/messages
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r219476048
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Spark deploy mode of Client
+
+To start STS in client mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>
+
+### Spark deploy mode of Cluster
+
+To start STS in cluster mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+ --deploy-mode cluster
+
+The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name incase of client
--- End diff --
pod/container from which the STS command is executed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r218950374
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Spark deploy mode of Client
+
+To start STS in client mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>
+
+### Spark deploy mode of Cluster
+
+To start STS in cluster mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+ --deploy-mode cluster
+
+The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name incase of client
--- End diff --
What you mean by `self pod name`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 closed the pull request at:
https://github.com/apache/spark/pull/22433
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
Can some body pls merge this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> If possible, there should be some basic integration testing. Run a thrift server command against the minishift cluster used by the other testing.
Will add this a separate PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: Support STS to run in k8s deployment modes with spark de...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22433
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/22433
It is an implementation detail of k8s integration that application name is expected to be DNS compliant ... spark does not have that requirement; and yarn/mesos/standalone/local work without this restriction.
The right fix in k8s integration would be to sanitize the name specified by user/application to be compliant with its requirements. This will help not just with thrift server, but any spark application.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22433#discussion_r218950593
--- Diff: docs/running-on-kubernetes.md ---
@@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl
[Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and
[Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/).
+## Running Spark Thrift Server
+
+Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQL’s port of Apache Hive’s HiveServer2 that allows
+JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark.
+
+### Spark deploy mode of Client
+
+To start STS in client mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>
+
+### Spark deploy mode of Cluster
+
+To start STS in cluster mode, excute the following command
+
+$ sbin/start-thriftserver.sh \
+ --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
+ --deploy-mode cluster
+
+The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name incase of client
+mode, which can be found with kubectl get pods), and run kubectl port-forward spark-app-podname 31416:10000
--- End diff --
Can we wrap commands into a pair of ``?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> Agreed with @mridulm that the naming restriction is specific to k8s and should be handled in a k8s specific way, e.g., somewhere around https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L208.
Ok, Will update the PR with the same.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by erikerlandson <gi...@git.apache.org>.
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/22433
I'm wondering, is there some reason this isn't supported in cluster mode for yarn & mesos? Or put another way, what is the rationale for k8s being added as an exception to this rule?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> It is an implementation detail of k8s integration that application name is expected to be DNS compliant ... spark does not have that requirement; and yarn/mesos/standalone/local work without this restriction.
> The right fix in k8s integration would be to sanitize the name specified by user/application to be compliant with its requirements. This will help not just with thrift server, but any spark application.
As this script is common start point for all the resource managers(k8s/yarn/mesos/standalone/local), i guess changing this to fit for all the cases has a value add, instead of doing at each resource manager level. Thoughts?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by suryag10 <gi...@git.apache.org>.
Github user suryag10 commented on the issue:
https://github.com/apache/spark/pull/22433
> I'm wondering, is there some reason this isn't supported in cluster mode for yarn & mesos? Or put another way, what is the rationale for k8s being added as an exception to this rule?
I donno the specific reason why this was not supported in yarn and mesos. The initial contributions to the spark on K8S started with cluster mode(with restriction for client mode). So this PR enhances such that STS can run in k8s deployments with spark cluster mode(In the latest spark code i had observed that the client mode also works(need to cross verify this once)).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by erikerlandson <gi...@git.apache.org>.
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/22433
If possible, there should be some basic integration testing. Run a thrift server command against the minishift cluster used by the other testing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by erikerlandson <gi...@git.apache.org>.
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/22433
In the scenario of a cluster-mode submission, what is the command-line behavior? Does the thrift-server script "block" until the thrift server pod is shut down?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s deploy...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/22433
Agreed with @mridulm that the naming restriction is specific to k8s and should be handled in a k8s specific way, e.g., somewhere around https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L208.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org