You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2020/08/10 12:37:43 UTC
[GitHub] [shardingsphere-elasticjob] zhaoyuguang opened a new issue #1355: Support Kafka job
zhaoyuguang opened a new issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355
I want create new maven module for elasticjob-kafka-executor under elasticjob-executor-type to implement new job type for Kafka.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] zhaoyuguang commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
zhaoyuguang commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-683302722
Compared to some other job types, it does not need to worry about blocking and execution time. In this scenario, a kafka Job Type Elastic Job project could complete the task completely in company.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] rudy2steiner commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
rudy2steiner commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-671696096
great job, I'm interested in when and where we will use MQ job? maybe I can help
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-671367597
It is a good idea, we can do more built-in job types
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] zhaoyuguang commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
zhaoyuguang commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-695753231
I want to finish this problem in several stages.
Stage 1: New Maven Module for Elasticjob-Kafka-Executor Under Elasticjob-Executor Type to Implement New Job Type for Kafka
Phase 2: Jar jar conflicts are avoided by using the Maven-Shade Plugin
Stage 3: Improve relevant documents
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] zhaoyuguang edited a comment on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
zhaoyuguang edited a comment on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-683302722
Compared to some other job types, it does not need to worry about blocking and execution time as https://github.com/apache/shardingsphere-elasticjob/issues/1405. In this scenario, a kafka Job Type Elastic Job project could complete the task completely in company.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] zhaoyuguang commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
zhaoyuguang commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-671692791
> How about design a MQ Job, and use SPI to load Kafka, RocketMQ and Pulsar?
This is the best one, but it's a bit difficult to implement at once. I want to create `elasticJob-kafka-executor` first and then use SPI load Kafka and other MQ. Is that ok?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-680969699
What is benefit for use ElasticJob to produce message？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] zhaoyuguang edited a comment on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
zhaoyuguang edited a comment on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-678613290
1 When multiple consumers listen to the same topic, they can execute tasks at the same time and complete their own business code
2 Task execution is asynchronous and does not block
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] zhaoyuguang edited a comment on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
zhaoyuguang edited a comment on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-678613290
1 When multiple consumers listen to the same topic, they can execute tasks at the same time and complete their own business code
2 Task execution is asynchronous and does not block
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-671690979
How about design a MQ Job, and use SPI to load Kafka, RocketMQ and Pulsar?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] rudy2steiner edited a comment on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
rudy2steiner edited a comment on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-671696096
great idea, I'm interested in when and where we will use MQ job? maybe I can help
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] zhaoyuguang commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
zhaoyuguang commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-671336113
I wanted to do this with Kafka-client to increase the types of tasks we support, similar to HTTP jobs, but the user had to introduce the kafka-client package, and if they didn't want to, they had to use the maven excluded tag to excluded `elasticjob-kafka-executor`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] zhaoyuguang commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
zhaoyuguang commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-674737563
Support for more job types
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-674492663
We need to discuss what is the best practice if using ElasticJob for improving. Can you provide some hint of it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] zhaoyuguang commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
zhaoyuguang commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-678613290
[diagram](https://user-images.githubusercontent.com/10150229/90952179-1feadd80-e494-11ea-9f1d-d263baec2179.png)
1 When multiple consumers listen to the same topic, they can execute tasks at the same time and complete their own business code
2 Task execution is asynchronous and does not block
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-678605330
Can we discuss more details about the usage of Kafka Job?
Cloud you provide a diagram to describe your idea?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] Technoboy- commented on issue #1355: Support Kafka job
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #1355:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1355#issuecomment-671701757
So could we discuss about the scenarios that involved in MQ?
1. We use mq-executor to send a trigger event to the target broker and topic as the job triggered or finished ？ This seems a cool ideal.
I can only think about the mq-executor as a producer, but not a consumer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org