You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by "ChengJie1053 (via GitHub)" <gi...@apache.org> on 2023/07/13 07:33:54 UTC
[GitHub] [linkis] ChengJie1053 opened a new pull request, #4767: Spark supports k8s operator submit task
ChengJie1053 opened a new pull request, #4767:
URL: https://github.com/apache/linkis/pull/4767
Spark supports k8s operator submit task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on a diff in pull request #4767: Spark supports k8s operator submit task
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on code in PR #4767:
URL: https://github.com/apache/linkis/pull/4767#discussion_r1267464066
##########
linkis-engineconn-plugins/spark/src/main/resources/spark-k8s-operator.md:
##########
@@ -0,0 +1,98 @@
+
+### 1. spark-on-k8s-operator document
+
+```text
+https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/quick-start-guide.md
+```
+
+
+### 2. spark-on-k8s-operator install
+
+```text
+helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator
+
+helm install my-release spark-operator/spark-operator --namespace spark-operator --create-namespace --set webhook.enable=true
+```
+
+### 3. spark-on-k8s-operator test task submit
+
+```text
+kubectl apply -f examples/spark-pi.yaml
+```
+
+### 4. If an error is reported: Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-pi-driver" is forbidden: error looking up service account spark/spark: serviceaccount "spark" not found.
+
+```text
+kubectl create serviceaccount spark
+
+kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
+```
+
+### 5. spark-on-k8s-operator Uninstall (usually not required, uninstall after installation problems)
+
+```text
+helm uninstall my-release --namespace spark-operator
+
+kubectl delete serviceaccounts my-release-spark-operator --namespace spark-operator
+
+kubectl delete clusterrole my-release-spark-operator --namespace spark-operator
+
+kubectl delete clusterrolebindings my-release-spark-operator --namespace spark-operator
+```
+
+### 6. Submitting tasks with Restful API
+```text
+POST /api/rest_j/v1/entrance/submit
Review Comment:
need to add linkis-cli demo
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/factory/SparkEngineConnFactory.scala:
##########
@@ -95,6 +95,17 @@ class SparkEngineConnFactory extends MultiExecutorEngineConnFactory with Logging
sparkConfig.setJavaHome(variable(Environment.JAVA_HOME))
sparkConfig.setSparkHome(SPARK_HOME.getValue(options))
sparkConfig.setMaster(SPARK_MASTER.getValue(options))
+ sparkConfig.setK8sConfigFile(SPARK_K8S_CONFIG_FILE.getValue(options))
Review Comment:
Is there a need to judge Yarn or k8s here?
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/config/SparkConfiguration.scala:
##########
@@ -50,6 +50,18 @@ object SparkConfiguration extends Logging {
val SPARK_APP_RESOURCE = CommonVars[String]("spark.app.resource", "")
val SPARK_APP_CONF = CommonVars[String]("spark.extconf", "")
+ val SPARK_K8S_CONFIG_FILE = CommonVars[String]("spark.k8s.config.file", "")
Review Comment:
If it is not a native parameter of spark or k8s, it need to add links prefix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] chenghuichen commented on pull request #4767: Spark supports k8s operator submit task
Posted by "chenghuichen (via GitHub)" <gi...@apache.org>.
chenghuichen commented on PR #4767:
URL: https://github.com/apache/linkis/pull/4767#issuecomment-1640569732
After all, `org.apache.linkis.engineplugin.spark.utils.SparkJobProgressUtil#getSparkJobInfo` should also be adapted for this submitting way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] ChengJie1053 commented on pull request #4767: Spark supports k8s operator submit task
Posted by "ChengJie1053 (via GitHub)" <gi...@apache.org>.
ChengJie1053 commented on PR #4767:
URL: https://github.com/apache/linkis/pull/4767#issuecomment-1641217501
> Besides, it would be nice to adapt `org.apache.linkis.engineplugin.spark.utils.SparkJobProgressUtil#getSparkJobInfo` to this submitting way.
Ok, thanks for reviewing the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on pull request #4767: Spark supports k8s operator submit task
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on PR #4767:
URL: https://github.com/apache/linkis/pull/4767#issuecomment-1636807308
ping @chenghuichen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] ChengJie1053 commented on a diff in pull request #4767: Spark supports k8s operator submit task
Posted by "ChengJie1053 (via GitHub)" <gi...@apache.org>.
ChengJie1053 commented on code in PR #4767:
URL: https://github.com/apache/linkis/pull/4767#discussion_r1267471937
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/config/SparkConfiguration.scala:
##########
@@ -50,6 +50,18 @@ object SparkConfiguration extends Logging {
val SPARK_APP_RESOURCE = CommonVars[String]("spark.app.resource", "")
val SPARK_APP_CONF = CommonVars[String]("spark.extconf", "")
+ val SPARK_K8S_CONFIG_FILE = CommonVars[String]("spark.k8s.config.file", "")
Review Comment:
Ok, thanks for reviewing the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong merged pull request #4767: Spark supports k8s operator submit task
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong merged PR #4767:
URL: https://github.com/apache/linkis/pull/4767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org