You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/05/25 03:57:08 UTC
[GitHub] [incubator-seatunnel] dijiekstra opened a new issue, #1947: [Discuss] Add Server for seatunnel
dijiekstra opened a new issue, #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### Describe the proposal
Because my English is not very good, so I post in Chinese first; I will add English instructions later.
## 背景
> 为什么我们需要一个Server
假设我现在是一个seatunnel的用户，我现在想要将数据库或者业务日志导入到OLAP引擎中。我现在只能通过命令行的方式进行任务提交，并且任务的停止&运维需要依赖于spark/flink；这给我们带来了巨大的额外工作量
1. 任务脚本的管理，如何对散落在服务器上的脚本进行管理？如果需要定时同步，那只能通过crontab完成；
2. 用户的提交和运维的入口不统一；
3. 在任务提交的前后，我可能需要前置或者后置处理，当然我也可以自己封装一层脚本，如果我每个任务都需要这样的处理，那么任务的管理会变得更加复杂；
4. 当然，我可以使用azkaban或者dolphinscheduler来完成上述的功能，但是这样引入了更多的组件，而我一开始就只是想将数据同步到OLAP中，我不太需要调度引擎的调度侧的能力，我仅仅是想更好的管理和运维我的seatunnel脚本；而且azkaban或dolphinscheduler上的能力，可能不足以支撑我们数据集成的场景，毕竟它们不是专门针对于数据集成这个场景。
回到seatunnel的开发者角度上来看
作为一个平台，一个服务，不提供可视化的管控平台，只提供命令行交互方式，那就是耍流氓。
管控平台需要做什么？
最主要的是管理数据集成任务的配置信息。
让用户通过WebUI能够轻松的完成任务的配置信息：比如输入&输出数据源、宇段信息、分区信息、过滤条件、异常数据处理、调度时间、并发度控制、流量控制、增量或全量配置等等
总之，就是尽量让用户能够通过配置信息来表达自己的业务诉求。
除了更好的开发，剩下的就是让运维变得更简单：
- 提供任务执行流水，便于用户查询任务执行情况；
- 提供数据源和权限的管理，这在多用户和多租户的场景下非常常见
- 提供系统的负载与监控、任务执行告警等
当然，这些能力理想状态还是能够与其他类型的作业整合到同一个平台上去(因为其余的作业也有相似的需求)，所以这里对管控平台的设计就有更高的要求：能够复用调度系统或者运维中心已有的能力，如果没有对应的服务，那么自己也应该有一套内置的能力来支撑这样的事情。
但是有一些事情，依赖于别的应用是做不了，或者说ROI太低
1. schema evolution：当然，这更依赖于引擎的能力，但是如果有管控平台，我们可以做一点简单化的`SE`，比如自动新增字段、将删减字段以空数据插入等，还是可行的
2. 数据时间：举一个例子，在mysql binlog -> hive 的场景，数据可能因为各种原因最后延迟到达了，那么如何将数据划分到正确的分区？
3. 动态分区：在某些场景，我们需要将历史数据重新同步一份到大数据集群，数据中已经有对应的分区字段，我们需要依赖这个字段插入到指定hive分区中，这在批处理中很容易就可以做到，但是在数据集成中，我们需要很大的代价，比如修改或新增spark/flink连接器。但如果我们有管控平台，我们提供`后置处理`的能力，那只需要seatunnel将数据写到一张临时表的分区中，然后再通过hive/spark/flink的批处理，即可完成对应的操作
所以，综上所述，为了用户更好的去使用seatunnel，一个管控平台对于我们来说是必不可少的。
## 目标
//todo
## 概要设计
//todo
## 详细设计
//todo
## 后续规划
//todo
### Task list
- [design] #1
- [Architectural Design] #2
- [Detail Design] #2
// i will fill task list soon
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1140615290
I'm not sure about we need `authority management`, `task alert` cause seatunnel is a framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] luketalent commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
luketalent commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1168350468
@dijiekstra I wanna join, how do I start??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1140274584
Hi, every contributors or others:
Would u like to work with me on this feature? I need your help. If I was developing by myself, it would take me a lot of time. After all, I have work to do, and I'm not doing open source full-time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1170846418
> Is ST not considering the function of theme switching?
Ignore these features for now. Thanks for your idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1142145297
Due to no one has questioned it, I will start development next week. I will update the progress in this issule regularly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1140641859
> I'm not sure about we need `authority management`, `task alert` cause seatunnel is a framework.
`task alert` depends on scheduler such as dolphinscheduler or others.
If we want to do it, we must check task status on realtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1140627969
> I'm not sure about we need `authority management`, `task alert` cause seatunnel is a framework.
It depends on the community, if SeaTunnel devotes to a platform, this could be good. to be clear TASK-Alert is usually handed over to the scheduling tool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] jianneng-fit2cloud commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by "jianneng-fit2cloud (via GitHub)" <gi...@apache.org>.
jianneng-fit2cloud commented on issue #1947:
URL: https://github.com/apache/seatunnel/issues/1947#issuecomment-1622927155
目前在用的时候有问题呀，No matched script save dir [/dj]，这个不知道怎么处理
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1161687712
ONLINE API VIEW
URL: https://www.apifox.cn/apidoc/shared-c6e2e561-8e6d-446d-9386-1c4a2c3ab50f PASSWORD : seatunnel-1947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1147371454
> This is a very good proposal. I think web server is a very important feature of SeaTunnel. Could you update the content to English?
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhuangchong commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
zhuangchong commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1210078543
What tasks are currently unclaimed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1172338892
The basic script management is already, I'll focus on the development of integration with Scheduler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1140642451
>
Or Scheduler push result to us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] luketalent commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
luketalent commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1146817415
good ！！ I'm in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] quanzhian commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
quanzhian commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1166180567
I am developing the back-end service interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1137121941
> This is a very good proposal. I think web server is a very important feature of SeaTunnel. Could you update the content to English?
Thank you for your comment.
I will translate the content to english after all design finish ,it'll be completed before the end of this week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] fanyo190 commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
fanyo190 commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1139321366
@dijiekstra 您说的功能什么时候发布呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1140274299
@CalvinKirs @ruanwenjun @gaojun2048
Hi, I have completed most of the design content, please help me review it .
If there is no problem, I will develop based on this design
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1140745016
>
Integration with Ranger is a good idea.
But I haven't connected Ranger, it'll take longer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dijiekstra commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
dijiekstra commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1140151556
> @dijiekstra 您说的功能什么时候发布呢？
It is still in the design stage and such a large feature needs to be discussed and approved by the community before it can be developed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] songjianet commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
songjianet commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1168870151
I will handle the work of the front-end part accordingly, please refer to the changes in the front-end part https://github.com/apache/incubator-seatunnel/issues/2076.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hxssssss commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
Hxssssss commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1168851293
@dijiekstra I wanna join, how do I start??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Tangsonghuai commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by "Tangsonghuai (via GitHub)" <gi...@apache.org>.
Tangsonghuai commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1522847234
现在什么进度了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1154634548
> [OnlineMeeting&June 7]SeaTunnel community meeting Topic collect #1986
We are looking for people who are willing to work together for this feature. Are you interested in participating？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] mengfeiMonica commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
mengfeiMonica commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1156083547
Hi guys , My name is Monica and I am a PM.I designed some parts of function below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] songjianet commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
songjianet commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1169453526
Is ST not considering the function of theme switching?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 2013650523 commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
2013650523 commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1193283167
@dijiekstra I wanna join, how do I start??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] gaojun2048 commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
gaojun2048 commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1136944636
This is a very good proposal. I think web server is a very important feature of SeaTunnel.
Could you update the content to English?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] legendtkl commented on issue #1947: [Discuss] Add Server for seatunnel
Posted by GitBox <gi...@apache.org>.
legendtkl commented on issue #1947:
URL: https://github.com/apache/incubator-seatunnel/issues/1947#issuecomment-1140743543
Great. This proposal will be a long building.
I suggest to take the integration into consideration, such as user integration with LDAP, auth integration with Ranger, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org