You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/04/26 09:28:24 UTC
[GitHub] [dolphinscheduler] sq-q opened a new pull request, #9789: [Feature-9772][plugin/ui] spark sql support for SparkTask
sq-q opened a new pull request, #9789:
URL: https://github.com/apache/dolphinscheduler/pull/9789
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
According to user requirements, we hope to support spark sql for SparkTask.
Users want to have an editor to fill in the sql script, which is convenient for ETL.
The renderings we need are as follows:
## Brief change log
- ProgramType.class add sql type
- SparkTask.class and SparkArgsUtils.class add main logical for support Spark sql command
- use-spark.ts adds some logic to support front-end pages supporting spark sql editing SQL
## Verify this pull request
This change added tests and can be verified as follows:
org.apache.dolphinscheduler.plugin.task.spark.SparkTaskTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sq-q commented on pull request #9789: [Feature-9772][plugin/ui] spark sql support for SparkTask
Posted by GitBox <gi...@apache.org>.
sq-q commented on PR #9789:
URL: https://github.com/apache/dolphinscheduler/pull/9789#issuecomment-1109610895
> Hi @sq-q , please check failed CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #9789: [Feature-9772][plugin/ui] spark sql support for SparkTask
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #9789:
URL: https://github.com/apache/dolphinscheduler/pull/9789#discussion_r858500222
##########
dolphinscheduler-common/src/main/resources/common.properties:
##########
@@ -92,4 +92,4 @@ development.state=false
alert.rpc.port=50052
# Url endpoint for zeppelin RESTful API
-zeppelin.rest.url="http://localhost:8080"
+zeppelin.rest.url="http://localhost:8080"
Review Comment:
We should avoid removing the last line.
##########
dolphinscheduler-master/src/main/resources/logback-spring.xml:
##########
@@ -72,4 +72,4 @@
<appender-ref ref="TASKLOGFILE"/>
<appender-ref ref="MASTERLOGFILE"/>
</root>
-</configuration>
+</configuration>
Review Comment:
Same here.
##########
dolphinscheduler-tools/src/main/resources/application.yaml:
##########
@@ -43,4 +43,4 @@ spring:
on-profile: mysql
datasource:
driver-class-name: com.mysql.jdbc.Driver
- url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
+ url: jdbc:mysql://localhost/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
Review Comment:
Why did you change this?
##########
dolphinscheduler-ui-next/src/views/projects/task/components/node/fields/use-spark.ts:
##########
@@ -136,4 +164,4 @@ export const SPARK_VERSIONS = [
label: 'SPARK1',
value: 'SPARK1'
}
-]
+]
Review Comment:
Same here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sq-q commented on a diff in pull request #9789: [Feature-9772][plugin/ui] spark sql support for SparkTask
Posted by GitBox <gi...@apache.org>.
sq-q commented on code in PR #9789:
URL: https://github.com/apache/dolphinscheduler/pull/9789#discussion_r858511908
##########
dolphinscheduler-tools/src/main/resources/application.yaml:
##########
@@ -43,4 +43,4 @@ spring:
on-profile: mysql
datasource:
driver-class-name: com.mysql.jdbc.Driver
- url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
+ url: jdbc:mysql://localhost/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8
Review Comment:
I am modifying
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sq-q closed pull request #9789: [Feature-9772][plugin/ui] spark sql support for SparkTask
Posted by GitBox <gi...@apache.org>.
sq-q closed pull request #9789: [Feature-9772][plugin/ui] spark sql support for SparkTask
URL: https://github.com/apache/dolphinscheduler/pull/9789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sq-q commented on a diff in pull request #9789: [Feature-9772][plugin/ui] spark sql support for SparkTask
Posted by GitBox <gi...@apache.org>.
sq-q commented on code in PR #9789:
URL: https://github.com/apache/dolphinscheduler/pull/9789#discussion_r858511322
##########
dolphinscheduler-common/src/main/resources/common.properties:
##########
@@ -92,4 +92,4 @@ development.state=false
alert.rpc.port=50052
# Url endpoint for zeppelin RESTful API
-zeppelin.rest.url="http://localhost:8080"
+zeppelin.rest.url="http://localhost:8080"
Review Comment:
I am changing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on pull request #9789: [Feature-9772][plugin/ui] spark sql support for SparkTask
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #9789:
URL: https://github.com/apache/dolphinscheduler/pull/9789#issuecomment-1109579732
Hi @sq-q , please check failed CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org