You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/07/20 08:52:43 UTC
[GitHub] [incubator-kyuubi] yaooqinn opened a new pull request #841: Kyuubi Spark External Shuffle Service
yaooqinn opened a new pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
#### Kyuubi Spark External Shuffle Service
This enables the missing feature of Dynamic Resource Allocation(a.k.a, DRA, DA) for running Spark on Kubernetes, which provides the auto-scaling capability for Spark application, especially downscaling.
Although there is a feature called shuffle tracking to track the shuffle data and tell Spark to remove executor while the data exceeds the time to live, these removals are coarse-grained.
##### Prerequisites
- Everything you need to know about Spark on Kubernetes
- Running Spark on Kubernetes apps with hostNetwork enabled
- Running this module as a DaemonSet with hostNetwork enabled
- Self-managed shuffle data cleaner as the state of apps will not be posted to this service
- See: tools/spark-block-cleaner/kubernetes/spark-block-cleaner.yml
##### Usage
Check `tools/kyuubi-shuffle-service/kubernetes/example.yaml` for more information.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] holdenk commented on pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#issuecomment-884364030
I like the simplicity of this approach, but I think there are some pretty serious limitations we should think through. It would let us scale up and down executor pods but not the shuffle services and we'd probably end up doing a lot of remote fetches. Can you give me some more context around your plans here? Have you looked at the current work on introducing the new shuffle APIs for Kube in Spark?
Is the goal just to support dynamic scaling? If so have you tried enabling migrations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#discussion_r672935949
##########
File path: tools/kyuubi-shuffle-service/kubernetes/example.yaml
##########
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# kubectl apply -f tools/kyuubi-shuffle-service/kubernetes/example.yaml
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: kyuubi-shuffle-service
+ namespace: default
+spec:
+ selector:
+ matchLabels:
+ name: kyuubi-shuffle-service
+ template:
+ metadata:
+ labels:
+ name: kyuubi-shuffle-service
+ spec:
+ containers:
+ # the image built base on ./docker/Dockerfile
+ - image: yaooqinn/kyuubi-shuffle-service:latest
+ name: exterbal-shuffle-server
+ resources:
+ limits:
+ memory: 1024Mi
+ requests:
+ cpu: 100m
+ memory: 512Mi
+ volumeMounts:
+ - name: spark-local-dir-1
+ mountPath: /opt/spark/data1
+ - name: spark-local-dir-2
+ mountPath: /opt/spark/data2
+ env:
+ # Change this in case of OOM based on you spec.containers.resources.* for
+ # the heap memory of shuffle service proc
+ - name: SPARK_DAEMON_MEMORY
+ value: 512M
+ # SPARK_SHUFFLE_OPTS allows us to pass spark configurations to the shuffle service
+ # spark.local.dir points to spec.containers.volumeMounts.*, shall be same as Spark
+ # app side.
+ # spark.shuffle.cleaner.interval=30s, this currently not work because only can a
Review comment:
If an `ExternalBlockStoreClient` can also post ShuffleServiceHeartbea, we can reach much more fine grained disk/shuffle data management
##########
File path: tools/kyuubi-shuffle-service/kubernetes/example.yaml
##########
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# kubectl apply -f tools/kyuubi-shuffle-service/kubernetes/example.yaml
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: kyuubi-shuffle-service
+ namespace: default
+spec:
+ selector:
+ matchLabels:
+ name: kyuubi-shuffle-service
+ template:
+ metadata:
+ labels:
+ name: kyuubi-shuffle-service
+ spec:
+ containers:
+ # the image built base on ./docker/Dockerfile
+ - image: yaooqinn/kyuubi-shuffle-service:latest
+ name: exterbal-shuffle-server
+ resources:
+ limits:
+ memory: 1024Mi
+ requests:
+ cpu: 100m
+ memory: 512Mi
+ volumeMounts:
+ - name: spark-local-dir-1
+ mountPath: /opt/spark/data1
+ - name: spark-local-dir-2
+ mountPath: /opt/spark/data2
+ env:
+ # Change this in case of OOM based on you spec.containers.resources.* for
+ # the heap memory of shuffle service proc
+ - name: SPARK_DAEMON_MEMORY
+ value: 512M
+ # SPARK_SHUFFLE_OPTS allows us to pass spark configurations to the shuffle service
+ # spark.local.dir points to spec.containers.volumeMounts.*, shall be same as Spark
+ # app side.
+ # spark.shuffle.cleaner.interval=30s, this currently not work because only can a
Review comment:
If an `ExternalBlockStoreClient` can also post ShuffleServiceHeartbeat, we can get much more fine-grained disk/shuffle data management
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#issuecomment-884239174
Thanks @holdenk
> So I don't know anything about this project but certainly does seem like it could be interesting.
Do worry about the project itself. The project relies on Spark who can achieve better scalability to use computing resources more effectively on Kubernetes. And this is the motivation of this PR.
> This provides an alternative shuffle service for Spark to use on Kube? Is there a design doc?
Yes, this provides an external shuffle service for Spark to use on Kube. The current implementation is codeless as you can see in this PR. We only need to write a Dockerfile(3 LOC only) for the shuffle service(`org.apache.spark.deploy.ExternalShuffleService`) endpoint based on the official Spark image and then deploy it on k8s as a DaemonSet. This can work with official releases(verified 3.1.2) without any modification. The only thing a bit tricky here is that users need to enable the `hostNetwork` to let the client successfully establish the connection `LOCAL` shuffle server during an executor initialization.
> I notice hostNetwork has to be set to true which is something I'm generally speaking not comfortable with for security reasons, is that a temporary design decision or something more permanent?
I agree that using hostNetwork is not secure and incomprehensive for this feature. I will try to figure out a solution with the Kubernetes pod network only, maybe in the next few weeks with a SPIP or a POC PR in the Spark community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#discussion_r672935949
##########
File path: tools/kyuubi-shuffle-service/kubernetes/example.yaml
##########
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# kubectl apply -f tools/kyuubi-shuffle-service/kubernetes/example.yaml
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: kyuubi-shuffle-service
+ namespace: default
+spec:
+ selector:
+ matchLabels:
+ name: kyuubi-shuffle-service
+ template:
+ metadata:
+ labels:
+ name: kyuubi-shuffle-service
+ spec:
+ containers:
+ # the image built base on ./docker/Dockerfile
+ - image: yaooqinn/kyuubi-shuffle-service:latest
+ name: exterbal-shuffle-server
+ resources:
+ limits:
+ memory: 1024Mi
+ requests:
+ cpu: 100m
+ memory: 512Mi
+ volumeMounts:
+ - name: spark-local-dir-1
+ mountPath: /opt/spark/data1
+ - name: spark-local-dir-2
+ mountPath: /opt/spark/data2
+ env:
+ # Change this in case of OOM based on you spec.containers.resources.* for
+ # the heap memory of shuffle service proc
+ - name: SPARK_DAEMON_MEMORY
+ value: 512M
+ # SPARK_SHUFFLE_OPTS allows us to pass spark configurations to the shuffle service
+ # spark.local.dir points to spec.containers.volumeMounts.*, shall be same as Spark
+ # app side.
+ # spark.shuffle.cleaner.interval=30s, this currently not work because only can a
Review comment:
If an `ExternalBlockStoreClient` can also post ShuffleServiceHeartbea, we can reach much more fine grained disk/shuffle data management
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#discussion_r672935949
##########
File path: tools/kyuubi-shuffle-service/kubernetes/example.yaml
##########
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# kubectl apply -f tools/kyuubi-shuffle-service/kubernetes/example.yaml
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: kyuubi-shuffle-service
+ namespace: default
+spec:
+ selector:
+ matchLabels:
+ name: kyuubi-shuffle-service
+ template:
+ metadata:
+ labels:
+ name: kyuubi-shuffle-service
+ spec:
+ containers:
+ # the image built base on ./docker/Dockerfile
+ - image: yaooqinn/kyuubi-shuffle-service:latest
+ name: exterbal-shuffle-server
+ resources:
+ limits:
+ memory: 1024Mi
+ requests:
+ cpu: 100m
+ memory: 512Mi
+ volumeMounts:
+ - name: spark-local-dir-1
+ mountPath: /opt/spark/data1
+ - name: spark-local-dir-2
+ mountPath: /opt/spark/data2
+ env:
+ # Change this in case of OOM based on you spec.containers.resources.* for
+ # the heap memory of shuffle service proc
+ - name: SPARK_DAEMON_MEMORY
+ value: 512M
+ # SPARK_SHUFFLE_OPTS allows us to pass spark configurations to the shuffle service
+ # spark.local.dir points to spec.containers.volumeMounts.*, shall be same as Spark
+ # app side.
+ # spark.shuffle.cleaner.interval=30s, this currently not work because only can a
Review comment:
If an `ExternalBlockStoreClient` can also post ShuffleServiceHeartbeat, we can get much more fine-grained disk/shuffle data management
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#discussion_r672935949
##########
File path: tools/kyuubi-shuffle-service/kubernetes/example.yaml
##########
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# kubectl apply -f tools/kyuubi-shuffle-service/kubernetes/example.yaml
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: kyuubi-shuffle-service
+ namespace: default
+spec:
+ selector:
+ matchLabels:
+ name: kyuubi-shuffle-service
+ template:
+ metadata:
+ labels:
+ name: kyuubi-shuffle-service
+ spec:
+ containers:
+ # the image built base on ./docker/Dockerfile
+ - image: yaooqinn/kyuubi-shuffle-service:latest
+ name: exterbal-shuffle-server
+ resources:
+ limits:
+ memory: 1024Mi
+ requests:
+ cpu: 100m
+ memory: 512Mi
+ volumeMounts:
+ - name: spark-local-dir-1
+ mountPath: /opt/spark/data1
+ - name: spark-local-dir-2
+ mountPath: /opt/spark/data2
+ env:
+ # Change this in case of OOM based on you spec.containers.resources.* for
+ # the heap memory of shuffle service proc
+ - name: SPARK_DAEMON_MEMORY
+ value: 512M
+ # SPARK_SHUFFLE_OPTS allows us to pass spark configurations to the shuffle service
+ # spark.local.dir points to spec.containers.volumeMounts.*, shall be same as Spark
+ # app side.
+ # spark.shuffle.cleaner.interval=30s, this currently not work because only can a
Review comment:
If an `ExternalBlockStoreClient` can also post ShuffleServiceHeartbea, we can reach much more fine grained disk/shuffle data management
##########
File path: tools/kyuubi-shuffle-service/kubernetes/example.yaml
##########
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# kubectl apply -f tools/kyuubi-shuffle-service/kubernetes/example.yaml
+
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: kyuubi-shuffle-service
+ namespace: default
+spec:
+ selector:
+ matchLabels:
+ name: kyuubi-shuffle-service
+ template:
+ metadata:
+ labels:
+ name: kyuubi-shuffle-service
+ spec:
+ containers:
+ # the image built base on ./docker/Dockerfile
+ - image: yaooqinn/kyuubi-shuffle-service:latest
+ name: exterbal-shuffle-server
+ resources:
+ limits:
+ memory: 1024Mi
+ requests:
+ cpu: 100m
+ memory: 512Mi
+ volumeMounts:
+ - name: spark-local-dir-1
+ mountPath: /opt/spark/data1
+ - name: spark-local-dir-2
+ mountPath: /opt/spark/data2
+ env:
+ # Change this in case of OOM based on you spec.containers.resources.* for
+ # the heap memory of shuffle service proc
+ - name: SPARK_DAEMON_MEMORY
+ value: 512M
+ # SPARK_SHUFFLE_OPTS allows us to pass spark configurations to the shuffle service
+ # spark.local.dir points to spec.containers.volumeMounts.*, shall be same as Spark
+ # app side.
+ # spark.shuffle.cleaner.interval=30s, this currently not work because only can a
Review comment:
If an `ExternalBlockStoreClient` can also post ShuffleServiceHeartbeat, we can get much more fine-grained disk/shuffle data management
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#issuecomment-883232465
cc @holdenk @dongjoon-hyun to see if we have an opportunity to make the current external shuffle service work for all cluster managers, including k8s, standalone, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#issuecomment-883232465
cc @holdenk @dongjoon-hyun to see if we have an opportunity to make the current external shuffle service work for all cluster managers, including k8s, standalone, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#issuecomment-883232465
cc @holdenk @dongjoon-hyun to see if we have an opportunity to make the current external shuffle service work for all cluster managers, including k8s, standalone, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] holdenk commented on pull request #841: Kyuubi Spark External Shuffle Service
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #841:
URL: https://github.com/apache/incubator-kyuubi/pull/841#issuecomment-883570132
So I don't know anything about this project but certainly does seem like it could be interesting. This provides an alternative shuffle service for Spark to use on Kube? Is there a design doc? I notice hostNetwork has to be set to true which is something I'm generally speaking not comfortable with for security reasons, is that a temporary design decision or something more permanent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org