You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampark.apache.org by Yuepeng Pan <pa...@apache.org> on 2024/04/02 02:50:22 UTC
Re: [DISCUSS] StreamPark Platform configuration files improvements
Hi, Ben.
Thank you for driving it.
Sounds good to me~
Best,
Yuepeng Pan
On 2024/03/30 10:07:42 Huajie Wang wrote:
> hi devs:
>
>
> Currently, the streampark platform provides multiple configuration
> files for user configuration, such as: application.yml,
> application-pgsql.yml, application-mysql.yml, kerberos.yml... , We can
> improve these configuration files. Many config files are internal
> system configurations, for example, in application.yml, a large number
> of configurations are internal platform configurations, such as
> jackson config for integration Spring Boot, swagger-ui config. the
> 'allow-circular-references' parameter for Spring... These do not need
> user configuration and should not be exposed to users.
>
> application.yml:
> ```yaml
>
> server:
> port: 10000
> undertow:
> buffer-size: 1024
> direct-buffers: true
> threads:
> io: 4
> worker: 20
>
> logging:
> level:
> root: info
>
> knife4j:
> enable: true
> basic:
> # basic authentication, used to access swagger-ui and doc
> enable: false
> username: admin
> password: streampark
>
> springdoc:
> api-docs:
> enabled: true
> swagger-ui:
> path: /swagger-ui.html
> packages-to-scan: org.apache.streampark.console
>
> spring:
> profiles.active: h2 #[h2,pgsql,mysql]
> application.name: StreamPark
> devtools.restart.enabled: false
> mvc.pathmatch.matching-strategy: ant_path_matcher
> servlet:
> multipart:
> enabled: true
> max-file-size: 500MB
> max-request-size: 500MB
> aop.proxy-target-class: true
> messages.encoding: utf-8
> jackson:
> date-format: yyyy-MM-dd HH:mm:ss
> time-zone: GMT+8
> deserialization:
> fail-on-unknown-properties: false
> main:
> allow-circular-references: true
> banner-mode: off
> mvc:
> converters:
> preferred-json-mapper: jackson
>
> management:
> endpoints:
> web:
> exposure:
> include: [ 'health', 'httptrace', 'metrics' ]
> endpoint:
> health:
> enabled: true
> show-details: always
> probes:
> enabled: true
> health:
> ldap:
> enabled: false
>
> streampark:
> proxy:
> # knox process address
> https://cdpsit02.example.cn:8443/gateway/cdp-proxy/yarn
> yarn-url:
> # lark alert proxy,default https://open.feishu.cn
> lark-url:
> yarn:
> # default simple, or kerberos
> http-auth: simple
>
> # HADOOP_USER_NAME
> hadoop-user-name: hdfs
> # local workspace, used to store source code and build dir etc.
> workspace:
> local: /opt/streampark_workspace
> remote: hdfs:///streampark # support hdfs:///streampark/ 、
> /streampark 、hdfs://host:ip/streampark/
>
> # remote docker register namespace for streampark
> docker:
> # instantiating DockerHttpClient
> http-client:
> max-connections: 10000
> connection-timeout-sec: 10000
> response-timeout-sec: 12000
> docker-host: ""
>
> # flink-k8s tracking configuration
> flink-k8s:
> tracking:
> silent-state-keep-sec: 10
> polling-task-timeout-sec:
> job-status: 120
> cluster-metric: 120
> polling-interval-sec:
> job-status: 2
> cluster-metric: 3
> # If you need to specify an ingress controller, you can use this.
> ingress:
> class: nginx
>
> # packer garbage resources collection configuration
> packer-gc:
> # maximum retention time for temporary build resources
> max-resource-expired-hours: 120
> # gc task running interval hours
> exec-cron: 0 0 0/6 * * ?
>
> shiro:
> # token timeout, unit second
> jwtTimeOut: 86400
> # backend authentication-free resources url
> anonUrl: >
>
> ldap:
> # Is ldap enabled? If so, please modify the urls
> enable: false
> ## AD server IP, default port 389
> urls: ldap://99.99.99.99:389
> ## Login Account
> base-dn: dc=streampark,dc=com
> username: cn=Manager,dc=streampark,dc=com
> password: streampark
> user:
> identity-attribute: uid
> email-attribute: mail
>
> ```
>
>
> So, I propose that we improve these configurations by providing users
> with only one configuration file(only one). The configurations in this
> file should be completely user-focused, clear, and core
> configurations.
>
> e.g:
> ```yaml
>
> # logging level
> logging.level.root: info
> # server port
> server.port: 10000
> # The user's login session has a validity period. If it exceeds this
> time, the user will be automatically logout
> # unit: s|m|h|d, s: second, m:minute, h:hour, d: day
> server.session.ttl: 2h # unit[s|m|h|d], e.g: 24h, 2d....
>
> # see: https://github.com/undertow-io/undertow/blob/master/core/src/main/java/io/undertow/Undertow.java
> server.undertow.direct-buffers: true
> server.undertow.buffer-size: 1024
> server.undertow.threads.io: 16
> server.undertow.threads.worker: 256
>
> # system database, default h2, mysql|pgsql|h2
> datasource.dialect: h2 # h2, pgsql
> #-------if datasource.dialect is mysql or pgsql, it is necessary to set-------
> datasource.username:
> datasource.password:
> # mysql jdbc url example:
> # datasource.url:
> jdbc:mysql://localhost:3306/streampark?useUnicode=true&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8
> # postgresql jdbc url example:
> # datasource.url:
> jdbc:postgresql://localhost:5432/streampark?stringtype=unspecified
> datasource.url:
> #---------------------------------------------------------------------------------
>
> # Directory for storing locally built project
> streampark.workspace.local: /tmp/streampark
> # The root hdfs path of the jars, Same as yarn.provided.lib.dirs for
> flink on yarn-application
> # and Same as --jars for spark on yarn
> streampark.workspace.remote: hdfs:///streampark/
> # hadoop yarn proxy path, e.g: knox process address
> https://streampark.com:8443/proxy/yarn
> streampark.proxy.yarn-url:
> # lark proxy address, default https://open.feishu.cn
> streampark.proxy.lark-url:
> # flink on yarn or spark on yarn, monitoring job status from yarn, it
> is necessary to set hadoop.http.authentication.type
> streampark.yarn.http-auth: simple # default simple, or kerberos
> # flink on yarn or spark on yarn, it is necessary to set
> streampark.hadoop-user-name: hdfs
> # flink on k8s ingress setting, If an ingress controller is specified
> in the configuration, the ingress class
> # kubernetes.io/ingress.class must be specified when creating the
> ingress, since there are often
> # multiple ingress controllers in a production environment.
> streampark.flink-k8s.ingress.class: nginx
>
> # sign streampark with ldap.
> ldap.enable: false # ldap enabled
> ldap.urls: ldap://99.99.99.99:389 #AD server IP, default port 389
> ldap.base-dn: dc=streampark,dc=com # Login Account
> ldap.username: cn=Manager,dc=streampark,dc=com
> ldap.password: streampark
> ldap.user.identity-attribute: uid
> ldap.user.email-attribute: mail
>
> # flink on yarn or spark on yarn, when the hadoop cluster enable
> kerberos authentication,
> # it is necessary to set up Kerberos authentication related parameters.
> security.kerberos.login.enable: false
> security.kerberos.login.debug: false
> # kerberos principal path
> security.kerberos.login.principal:
> security.kerberos.login.krb5:
> security.kerberos.login.keytab:
> security.kerberos.ttl: 2h # unit [s|m|h|d]
>
> ```
>
> this is issue: https://github.com/apache/incubator-streampark/issues/3641
>
> What's your opinion on this? Welcome to discuss
>
>
>
> Best,
> Huajie Wang
>