You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/31 11:27:52 UTC
[GitHub] [dolphinscheduler] limaiwang opened a new issue, #11720: [Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql
limaiwang opened a new issue, #11720:
URL: https://github.com/apache/dolphinscheduler/issues/11720
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql
### What you expected to happen
In spark-sql, selecting SPARK1 should execute ${SPARK_HOME1}/bin/spark-sql and selecting SPARK2 should execute ${SPARK_HOME1}/bin/spark-sql
### How to reproduce
![Uploading spark_sql.jpg…]()
[INFO] 2022-08-31 09:57:14.923 +0000 [taskAppId=TASK-20220831-6710611439552_3-40-44] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[69] - spark task params {"localParams":[],"rawScript":"SELECT * from ods.ods_pigeon_day LIMIT 1","resourceList":[],"programType":"SQL","mainClass":"","deployMode":"client","appName":"SparkSQLDemo","sparkVersion":"SPARK1","driverCores":1,"driverMemory":"512M","numExecutors":2,"executorMemory":"2G","executorCores":2}
[INFO] 2022-08-31 09:57:14.933 +0000 [taskAppId=TASK-20220831-6710611439552_3-40-44] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[240] - raw script : SELECT * from ods.ods_pigeon_day LIMIT 1
[INFO] 2022-08-31 09:57:14.934 +0000 [taskAppId=TASK-20220831-6710611439552_3-40-44] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[241] - task execute path : /tmp/dolphinscheduler/exec/process/6710440832960/6710611439552_3/40/44
[INFO] 2022-08-31 09:57:14.937 +0000 [taskAppId=TASK-20220831-6710611439552_3-40-44] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[130] - spark task command: ${SPARK_HOME2}/bin/spark-sql --master yarn --deploy-mode client --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --name SparkSQLDemo --queue haodf -f /tmp/dolphinscheduler/exec/process/6710440832960/6710611439552_3/40/44/40_44_node.sql
[INFO] 2022-08-31 09:57:14.937 +0000 [taskAppId=TASK-20220831-6710611439552_3-40-44] TaskLogLogger-class
### Anything else
_No response_
### Version
3.0.0
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #11720: [Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #11720:
URL: https://github.com/apache/dolphinscheduler/issues/11720#issuecomment-1237993997
Hi @limaiwang What about using the env management?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun closed issue #11720: [Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql
Posted by GitBox <gi...@apache.org>.
ruanwenjun closed issue #11720: [Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql
URL: https://github.com/apache/dolphinscheduler/issues/11720
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11720: [Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11720:
URL: https://github.com/apache/dolphinscheduler/issues/11720#issuecomment-1232812937
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] limaiwang commented on issue #11720: [Bug] [spark-sql] In spark-sql, select both SPARK1 and SPARK2 versions and execute ${SPARK_HOME2}/bin/spark-sql
Posted by GitBox <gi...@apache.org>.
limaiwang commented on issue #11720:
URL: https://github.com/apache/dolphinscheduler/issues/11720#issuecomment-1240055234
> Hi @limaiwang What about using the env management?
Hi @caishunfeng Are you trying to say that the spark-sql or spark-submit directories use the env mananget?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org