You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by "ChengJie1053 (via GitHub)" <gi...@apache.org> on 2023/06/09 03:32:07 UTC
[GitHub] [linkis] ChengJie1053 opened a new pull request, #4613: Spark supports k8s operator crd submit task
ChengJie1053 opened a new pull request, #4613:
URL: https://github.com/apache/linkis/pull/4613
Spark supports k8s operator crd submit task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] ChengJie1053 commented on a diff in pull request #4613: Spark supports k8s operator crd submit task
Posted by "ChengJie1053 (via GitHub)" <gi...@apache.org>.
ChengJie1053 commented on code in PR #4613:
URL: https://github.com/apache/linkis/pull/4613#discussion_r1226015027
##########
tool/dependencies/known-dependencies.txt:
##########
@@ -589,6 +614,7 @@ xmlbeans-5.1.1.jar
xmlenc-0.52.jar
xstream-1.4.20.jar
xz-1.5.jar
+zjsonpatch-0.3.0.jar
zookeeper-3.5.9.jar
zookeeper-jute-3.5.9.jar
Review Comment:
Ok, thanks for reviewing the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] ChengJie1053 closed pull request #4613: Spark supports k8s operator crd submit task
Posted by "ChengJie1053 (via GitHub)" <gi...@apache.org>.
ChengJie1053 closed pull request #4613: Spark supports k8s operator crd submit task
URL: https://github.com/apache/linkis/pull/4613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] casionone commented on a diff in pull request #4613: Spark supports k8s operator crd submit task
Posted by "casionone (via GitHub)" <gi...@apache.org>.
casionone commented on code in PR #4613:
URL: https://github.com/apache/linkis/pull/4613#discussion_r1224274846
##########
docs/configuration/spark-k8s-operator.md:
##########
@@ -0,0 +1,98 @@
+
+### 1. spark-on-k8s-operator document
+
+```text
+https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/quick-start-guide.md
+```
+
+
+### 2. spark-on-k8s-operator install
+
+```text
+helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator
+
+helm install my-release spark-operator/spark-operator --namespace spark-operator --create-namespace --set webhook.enable=true
+```
+
+### 3. spark-on-k8s-operator test task submit
+
+```text
+kubectl apply -f examples/spark-pi.yaml
+```
+
+### 4. If an error is reported: Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-pi-driver" is forbidden: error looking up service account spark/spark: serviceaccount "spark" not found.
+
+```text
+kubectl create serviceaccount spark
+
+kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
+```
+
+### 5. spark-on-k8s-operator Uninstall (usually not required, uninstall after installation problems)
+
+```text
+helm uninstall my-release --namespace spark-operator
+
+kubectl delete serviceaccounts my-release-spark-operator --namespace spark-operator
+
+kubectl delete clusterrole my-release-spark-operator --namespace spark-operator
+
+kubectl delete clusterrolebindings my-release-spark-operator --namespace spark-operator
+```
+
+### 6. Submitting tasks with Restful API
+```text
+POST /api/rest_j/v1/entrance/submit
+```
+
+```json
+{
+ "executionContent": {
+ "spark.app.main.class": "org.apache.spark.examples.SparkPi",
+ "spark.app.args": "spark.app.args",
+ "runType": "jar",
+ "code": "show databases"
+ },
+ "params": {
+ "variable": {
+ },
+ "configuration": {
+ "startup": {
+ "spark.executor.memory": "1g",
+ "spark.driver.memory": "1g",
+ "spark.executor.cores": "1",
+ "spark.app.name": "spark-submit-jar-cjtest",
+ "spark.app.resource": "local:///opt/spark/examples/jars/spark-examples_2.12-3.2.1.jar",
+ "spark.executor.instances": 1,
+ "spark.master": "k8soperator",
+ "spark.k8s.master.url": "http://ip:port",
+ "spark.k8s.username": "username",
+ "spark.k8s.password": "password"
+ }
+ }
+ },
+ "source": {
+ "scriptPath": "file:///tmp/hadoop/test.sql"
+ },
+ "labels": {
+ "engineType": "spark-3.2.1",
+ "engineConnMode": "once",
+ "userCreator": "linkis-IDE"
+ }
+}
+```
+
+### 7. Reference document
+```text
+https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
+
+https://github.com/fabric8io/kubernetes-client/
+
+https://github.com/apple/batch-processing-gateway
+
+https://www.lightbend.com/blog/how-to-manage-monitor-spark-on-kubernetes-introduction-spark-submit-kubernetes-operator
+
+https://www.lightbend.com/blog/how-to-manage-monitor-spark-on-kubernetes-deep-dive-kubernetes-operator-for-spark
+```
+
Review Comment:
According to this docs, I feel that it seems to be easy to run spark-on-k8s-operator module.
It may be possible to add as an integration test step（https://github.com/apache/linkis/blob/dev-1.4.0/.github/workflows/integration-test.yml） about spark-on-k8s-operator task later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] ChengJie1053 commented on pull request #4613: Spark supports k8s operator crd submit task
Posted by "ChengJie1053 (via GitHub)" <gi...@apache.org>.
ChengJie1053 commented on PR #4613:
URL: https://github.com/apache/linkis/pull/4613#issuecomment-1586461385
> please handle the third-party dependency error of the new jar package
Ok, thanks for reviewing the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] casionone commented on a diff in pull request #4613: Spark supports k8s operator crd submit task
Posted by "casionone (via GitHub)" <gi...@apache.org>.
casionone commented on code in PR #4613:
URL: https://github.com/apache/linkis/pull/4613#discussion_r1224254143
##########
tool/dependencies/known-dependencies.txt:
##########
@@ -589,6 +614,7 @@ xmlbeans-5.1.1.jar
xmlenc-0.52.jar
xstream-1.4.20.jar
xz-1.5.jar
+zjsonpatch-0.3.0.jar
zookeeper-3.5.9.jar
zookeeper-jute-3.5.9.jar
Review Comment:
need update license file at `linkis-dist/release-docs/licenses` for new dependencies
can refer to https://linkis.apache.org/zh-CN/docs/latest/development/development-specification/license#21-%E7%A4%BA%E4%BE%8B-%E5%9C%BA%E6%99%AF1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] aiceflower commented on pull request #4613: Spark supports k8s operator crd submit task
Posted by "aiceflower (via GitHub)" <gi...@apache.org>.
aiceflower commented on PR #4613:
URL: https://github.com/apache/linkis/pull/4613#issuecomment-1584216114
please handle the third-party dependency error of the new jar package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on pull request #4613: Spark supports k8s operator crd submit task
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on PR #4613:
URL: https://github.com/apache/linkis/pull/4613#issuecomment-1585574939
ping ~ @jacktao007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org