You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2020/11/13 07:07:40 UTC
[GitHub] [shardingsphere-elasticjob] johnny2002 opened a new issue #1720: Customization the JobInstance id
johnny2002 opened a new issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720
## Feature Request
For some reason, we need to customize the JobInstance ID, so that we can assign specific shard number to specific JobInstance.
And as I found in JobInstance.java, the JobInstance ID is fixed to IP concate with JVM name. The code as following:
```java
public JobInstance() {
jobInstanceId = IpUtils.getIp() + DELIMITER + ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
}
```
### Is your feature request related to a problem?
### Describe the feature you would like.
Suggest to check whether a "jobInstanceId" property is exists, if so, use IP concat with this property instead of jvm name.
code like this:
```java
public JobInstance() {
String jobInstIdSuffix = System.getProperty("jobInstanceId");
if (jobInstIdSuffix == null) {
jobInstIdSuffix = ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
}
jobInstanceId = IpUtils.getIp() + DELIMITER + jobInstIdSuffix;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727569831
> jobParameter
jobParameter can only identify usb-job, NOT jobInstance, namely, job executor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customization the JobInstance id
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-726568567
> You may submit a PR if you are interest in.
That's a good idea, I am doing it now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727673999
> > > > Detail reason for we need this feathure:
> > > > We have a large system that has large data, so we have to use unitized architecture.
> > > > So one job instance in specific unit can only execute specific sharding data.
> > > > e.g.
> > > > JobInstance-unit1 can only handle data sharding No.1
> > > > JobInstance-unit2 can only handle data sharding No.2
> > > > and so on
> > >
> > >
> > > As you said that your system is partition architecture, why not treat them as different jobs?
> >
> >
> > Then there will be 1000 jobs need to config one by one. and also hard to monitor.
>
> I don't quite understand your scenarios. Maybe you can just configure different job names while keeping other configuration consistent. I think configuring 1000 job names is as simple as configuring 1000 job instance id. Coorperating with Tracing module, you can immediately locate the problem by job name if any partition failed.
32 job names by 32 instance names, that's 1024 Job Instances, compare to configure 1024 job names, do you think which is simpler?
Another protential problem is that if we seperate one job to many jobs, then the configuration may be inconsistent.
Even though We can configuring 1000 job names, we still need to distinguish main and backup job instance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727908791
> > > This feature request is support static job assigned in ElasticJob which is not permit sub-jobs rebalance even job server crashed.
> >
> >
> > not exactly. we have jobinstances pair to backup each other.
> > Our complete sharding strategy sample:
> > JobInstance1 and JobInstance2 backup each other.
> > For sub-job1, firstly assign to instance1(main), if job instance1 crashed, then assign to instance2(backup)
> > For sub-job2, firstly assign to instance2(main), if job instance2 crashed, then assign to instance1(backup)
>
> What if 1 and 2 are both crashed?
Banking system usually has 3 data centers which located in 2 cities. Instance1 and Instance2 are located in 2 different centers, so very very low possibility for both crashed. If both crashed, then we will start to use center3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-728055492
> What I mean is we need to design a whole way to provide a project, not for patch only.
> Customized job instance id and sharding strategy together is not good for common user.
A graceful way is to add some customizable attributes in JobInstance, like group, tag, then put them in Zookeeper. Like Dubbo service does.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu commented on issue #1720: Customization the JobInstance id
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727515701
Hi, thank you for the suggestion.
I want discuss further.
Job instance id is internal impl in elastic job which user should not focus on them.
It is not a good idea to expose lots of internal impl to special reason.
Can you consider about reuse the job sharding strategy to rebalance your sub-job?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] TeslaCN commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727838069
> > This feature request is support static job assigned in ElasticJob which is not permit sub-jobs rebalance even job server crashed.
>
> not exactly. we have jobinstances pair to backup each other.
> Our complete sharding strategy sample:
> JobInstance1 and JobInstance2 backup each other.
> For sub-job1, firstly assign to instance1(main), if job instance1 crashed, then assign to instance2(backup)
> For sub-job2, firstly assign to instance2(main), if job instance2 crashed, then assign to instance1(backup)
What if 1 and 2 are both crashed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727673999
> > > > Detail reason for we need this feathure:
> > > > We have a large system that has large data, so we have to use unitized architecture.
> > > > So one job instance in specific unit can only execute specific sharding data.
> > > > e.g.
> > > > JobInstance-unit1 can only handle data sharding No.1
> > > > JobInstance-unit2 can only handle data sharding No.2
> > > > and so on
> > >
> > >
> > > As you said that your system is partition architecture, why not treat them as different jobs?
> >
> >
> > Then there will be 1000 jobs need to config one by one. and also hard to monitor.
>
> I don't quite understand your scenarios. Maybe you can just configure different job names while keeping other configuration consistent. I think configuring 1000 job names is as simple as configuring 1000 job instance id. Coorperating with Tracing module, you can immediately locate the problem by job name if any partition failed.
32 job names by 32 instance names, that's 1024 Job Instances, compare to configure 1024 job names, do you think which is simpler?
Another protential problem is that if we seperate one job to many jobs, then the configuration may be inconsistent.
Even though We can configure 1000 job names, we still need to distinguish main and backup job instance.
Our complete sharding strategy is that:
Firstly, assign sub-job1 to instance1(main), if job instance1 crashed, then assign to instance2(backup)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727560738
> @johnny2002 Hi, thank you for the suggestion.
> I want discuss further.
> Job instance id is internal impl in elastic job which user should not focus on them.
> It is not a good idea to expose lots of internal impl to special reason.
>
> Can you consider about reuse the job sharding strategy to rebalance your sub-job?
H Terry,
Of course I want to use my own sharding strategy to assign sub-job, while, the job strategy api passed in a list of instances, I need to exactly identify the jobinstance, so that I can assign corresponding sub-job to correct jobInstance. In my application, JobInstances are not equivalent. **Each jobinstance only has the resource to handle 1 or 2 specific sub-jobs.** e.g. JobInstance from unit 1 can only handle sub-job1 or sub-job2, CANNOT handle other sub-jobs.
Since JobInstance is exposed in JobShardingStrategy api, I don't think it is an internal impl.
My sharding strategy is:
sub-job1 assign to jobInstance 1(or you can say Jobexecutor 1)
sub-job2 assign to jobInstance 2
...
in JobShardingStrategy api, I need to identify each jobInstances due to they arenot equivalent.
```java
public Map<JobInstance, List<Integer>> sharding(final List<JobInstance> jobInstances, final String jobName, final int shardingTotalCount) {
```
My problem is that in my JobShardingStrategy impl, I have no way to know which JobInstance from which unit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727569644
> @johnny2002 Hi, thank you for the suggestion.
> I want discuss further.
> Job instance id is internal impl in elastic job which user should not focus on them.
> It is not a good idea to expose lots of internal impl to special reason.
>
> Can you consider about reuse the job sharding strategy to rebalance your sub-job?
Hi Terry,
My problem is cannot identify JobInstnaces.
If you don't like to change the jobinstance id, I have another graceful way like:
Pass all System Properties that name starts with "elasticjob." into JobInstance.
Sample code:
```java
public JobInstance() {
jobInstanceId = IpUtils.getIp() + DELIMITER + ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
//extract all system properties that name start with "elasticjob."
for (Map.Entry propEntry : System.getProperties().entrySet()) {
if (propEntry.getKey().toString().startsWith("elasticjob.")) {
properties.put((String)propEntry.getKey(), (String)propEntry.getValue());
}
}
}
```
Then we can identify JobInstance through JobInstance.properties
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] TeslaCN commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727572152
> > > Detail reason for we need this feathure:
> > > We have a large system that has large data, so we have to use unitized architecture.
> > > So one job instance in specific unit can only execute specific sharding data.
> > > e.g.
> > > JobInstance-unit1 can only handle data sharding No.1
> > > JobInstance-unit2 can only handle data sharding No.2
> > > and so on
> >
> >
> > As you said that your system is partition architecture, why not treat them as different jobs?
>
> Then there will be 1000 jobs need to config one by one. and also hard to monitor.
I don't quite understand your scenarios. Maybe you can just configure different job names while keeping other configuration consistents. I think configuring 1000 job names is as simple as configuring 1000 job instance id. Coorperating with Tracing module, you can immediately locate the problem by job name if any partition failed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-735735525
Hi Team,
How is the conclusion? Suggest to add group property to JobInstance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727836728
> This feature request is support static job assigned in ElasticJob which is not permit sub-jobs rebalance even job server crashed.
not exactly. we have jobinstances pair to backup each other.
Our complete sharding strategy sample:
JobInstance1 and JobInstance2 backup each other.
For sub-job1, firstly assign to instance1(main), if job instance1 crashed, then assign to instance2(backup)
For sub-job2, firstly assign to instance2(main), if job instance2 crashed, then assign to instance1(backup)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727569831
> Can you use jobParameter to recognize your customized sharding logic?
@terrymanu
jobParameter can only identify sub-job, NOT jobInstance or job executor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customization the JobInstance id
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-726568398
Detail reason for we need this feathure:
We have a large system that has large data, so we have to use unitized architecture.
So one job instance in specific unit can only execute specific sharding data.
e.g.
JobInstance-unit1 can only handle data sharding No.1
JobInstance-unit2 can only handle data sharding No.2
and so on
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727569644
> @johnny2002 Hi, thank you for the suggestion.
> I want discuss further.
> Job instance id is internal impl in elastic job which user should not focus on them.
> It is not a good idea to expose lots of internal impl to special reason.
>
> Can you consider about reuse the job sharding strategy to rebalance your sub-job?
Hi Terry,
My problem is cannot identify JobInstnaces. **Update on 2020-11-16: Sorry, I found this design not work, please ignore it.**
~~If you don't like to change the jobinstance id, I have another graceful way like:
Pass all System Properties that name starts with "elasticjob." into JobInstance.
Sample code:~~
```java
public final class JobInstance {
public static final String DELIMITER = "@-@";
private final String jobInstanceId;
private final Map<String, String> properties = new LinkedHashMap<>();
public JobInstance() {
jobInstanceId = IpUtils.getIp() + DELIMITER + ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
//extract all system properties that name start with "elasticjob."
for (Map.Entry propEntry : System.getProperties().entrySet()) {
if (propEntry.getKey().toString().startsWith("elasticjob.")) {
properties.put((String)propEntry.getKey(), (String)propEntry.getValue());
}
}
}
```
~~Then we can identify JobInstance through JobInstance.properties~~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727790982
This feature request is support static job assigned in ElasticJob which is not permit sub-jobs rebalance even job server crashed.
It is an interesting features. Open a SPI to permit user customize job instance id is good, but it is not enough, user still need to customize job sharding strategy.
Can we discuss the design to make a smart API to let user can use `static` ElasticJob?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727569831
> jobParameter
jobParameter can only identify sub-job, NOT jobInstance, namely, job executor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727560738
> @johnny2002 Hi, thank you for the suggestion.
> I want discuss further.
> Job instance id is internal impl in elastic job which user should not focus on them.
> It is not a good idea to expose lots of internal impl to special reason.
>
> Can you consider about reuse the job sharding strategy to rebalance your sub-job?
H Terry,
Of course I want to use my own sharding strategy to assign sub-job, while, the job strategy api passed in a list of instances, I need to exactly identify the jobinstance, so that I can assign corresponding sub-job to correct jobInstance. In my application, JobInstances are not equivalent. **Each jobinstance only has the resource to handle 1 or 2 specific sub-jobs.** e.g. JobInstance from unit 1 can only handle sub-job1 or sub-job2, CANNOT handle other sub-jobs.
Sine JobInstance is expose in JobShardingStrategy api, I don't think it is an internal impl.
My sharding strategy is:
sub-job1 assign to jobInstance 1(or you can say Jobexecutor 1)
sub-job2 assign to jobInstance 2
... ...
in JobShardingStrategy api, I need to identify each jobInstances.
```java
public Map<JobInstance, List<Integer>> sharding(final List<JobInstance> jobInstances, final String jobName, final int shardingTotalCount) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] TeslaCN edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
TeslaCN edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727572152
> > > Detail reason for we need this feathure:
> > > We have a large system that has large data, so we have to use unitized architecture.
> > > So one job instance in specific unit can only execute specific sharding data.
> > > e.g.
> > > JobInstance-unit1 can only handle data sharding No.1
> > > JobInstance-unit2 can only handle data sharding No.2
> > > and so on
> >
> >
> > As you said that your system is partition architecture, why not treat them as different jobs?
>
> Then there will be 1000 jobs need to config one by one. and also hard to monitor.
I don't quite understand your scenarios. Maybe you can just configure different job names while keeping other configuration consistent. I think configuring 1000 job names is as simple as configuring 1000 job instance id. Coorperating with Tracing module, you can immediately locate the problem by job name if any partition failed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727673999
> > > > Detail reason for we need this feathure:
> > > > We have a large system that has large data, so we have to use unitized architecture.
> > > > So one job instance in specific unit can only execute specific sharding data.
> > > > e.g.
> > > > JobInstance-unit1 can only handle data sharding No.1
> > > > JobInstance-unit2 can only handle data sharding No.2
> > > > and so on
> > >
> > >
> > > As you said that your system is partition architecture, why not treat them as different jobs?
> >
> >
> > Then there will be 1000 jobs need to config one by one. and also hard to monitor.
>
> I don't quite understand your scenarios. Maybe you can just configure different job names while keeping other configuration consistent. I think configuring 1000 job names is as simple as configuring 1000 job instance id. Coorperating with Tracing module, you can immediately locate the problem by job name if any partition failed.
32 job names by 32 instance names, that's 1024 Job Instances, compare to configure 1024 job names, do you think which is simpler?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727560738
> @johnny2002 Hi, thank you for the suggestion.
> I want discuss further.
> Job instance id is internal impl in elastic job which user should not focus on them.
> It is not a good idea to expose lots of internal impl to special reason.
>
> Can you consider about reuse the job sharding strategy to rebalance your sub-job?
H Terry,
Of course I want to use my own sharding strategy to assign sub-job, while, the job strategy api passed in a list of instances, I need to exactly identify the jobinstance, so that I can assign corresponding sub-job to correct jobInstance. In my application, JobInstances are not equivalent. **Each jobinstance only has the resource to handle 1 or 2 specific sub-jobs.** e.g. JobInstance from unit 1 can only handle sub-job1 or sub-job2, CANNOT handle other sub-jobs.
Sine JobInstance is expose in JobShardingStrategy api, I don't think it is an internal impl.
My sharding strategy is:
sub-job1 assign to jobInstance 1(or you can say Jobexecutor 1)
sub-job2 assign to jobInstance 2
... ...
in JobShardingStrategy api, I need to identify each jobInstances due to they arenot equivalent.
```java
public Map<JobInstance, List<Integer>> sharding(final List<JobInstance> jobInstances, final String jobName, final int shardingTotalCount) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727908791
> > > This feature request is support static job assigned in ElasticJob which is not permit sub-jobs rebalance even job server crashed.
> >
> >
> > not exactly. we have jobinstances pair to backup each other.
> > Our complete sharding strategy sample:
> > JobInstance1 and JobInstance2 backup each other.
> > For sub-job1, firstly assign to instance1(main), if job instance1 crashed, then assign to instance2(backup)
> > For sub-job2, firstly assign to instance2(main), if job instance2 crashed, then assign to instance1(backup)
>
> What if 1 and 2 are both crashed?
Banking system usually has 3 data centers which located in 2 remote places. Instance1 and Instance2 are located in 2 different centers, so very very low possibility for both crashed. If both crashed, then we will start to use center3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] TeslaCN edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
TeslaCN edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727567379
> Detail reason for we need this feathure:
>
> We have a large system that has large data, so we have to use unitized architecture.
> So one job instance in specific unit can only execute specific sharding data.
> e.g.
> JobInstance-unit1 can only handle data sharding No.1
> JobInstance-unit2 can only handle data sharding No.2
> and so on
As you said that your system is partition architecture, why not treat them as different jobs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-728055492
> What I mean is we need to design a whole way to provide a project, not for patch only.
> Customized job instance id and sharding strategy together is not good for common user.
A graceful way is to add some customizable attributes in JobInstance, like group, tag, etc. then put them in Zookeeper. Like Dubbo service does. e.g.
```java
@RequiredArgsConstructor
@Data
@EqualsAndHashCode(of = "jobInstanceId")
public final class JobInstance {
private static final String DELIMITER = "@-@";
private final String jobInstanceId;
private String group;
private String tag;
public JobInstance() {
jobInstanceId = IpUtils.getIp() + DELIMITER + ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
//TODO: Get value from spring environment is preferred
group = System.getProperty("elasticjob.jobInstance.group");
tag = System.getProperty("elasticjob.jobInstance.tag");
}
/**
* Get server IP address.
*
* @return server IP address
*/
public String getIp() {
return jobInstanceId.substring(0, jobInstanceId.indexOf(DELIMITER));
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727566366
Can you use jobParameter to recognize your customized sharding logic?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-726568398
Detail reason for we need this feathure:
We have a large system with large data, so we have to use unitized architecture.
So one job instance in specific unit can only execute specific sharding data.
e.g.
JobInstance-unit1 can only handle data sharding No.1
JobInstance-unit2 can only handle data sharding No.2
and so on
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] TeslaCN commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-736603659
> Hi Team,
>
> How is the conclusion? Suggest to add group property to JobInstance.
There is no conclusion yet. Fixing bugs is the first we need to do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] TeslaCN commented on issue #1720: Customization the JobInstance id
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-726568308
You may submit a PR if you are interest in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customization the JobInstance id
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727561219
Here is my sample strategy :
```java
/**
* @author WeiZhou
*
* shardingTotalCount should be as same as JobInstances count.
* Neighbor 2 JobInstances backup eachother.
* Each partition will be assigned to same Jobinstance number or its backup.
* shardingTotalCount和jobInstances数量要一样.
* 相邻两个JobInstances互相备份.
* 每个作业分片要分配给相同号码的JobInstance或是它的备份单元
*/
@Slf4j
public class UnitizedAllocationJobShardingStrategy implements JobShardingStrategy {
int[] backupMap = new int[] { 2, 1, 4, 3, 6, 5, 8, 7 };//Unit 1<>2, 3<>4, 5<>6, 7<>8 backup eachother
@Override
public Map<JobInstance, List<Integer>> sharding(final List<JobInstance> jobInstances, final String jobName, final int shardingTotalCount) {
if (jobInstances.isEmpty()) {
return Collections.emptyMap();
}
JobInstance[] instArray = new JobInstance[shardingTotalCount];
for (JobInstance jobInst : jobInstances) {
//Job instance IDs like xxx@-@001, xxx@-@002, xxx@-@003 ...
String jobInstanceId = jobInst.getJobInstanceId();
jobInstanceId = jobInstanceId.substring(jobInstanceId.indexOf(JobInstance.DELIMITER) + JobInstance.DELIMITER.length());
int jobInstNum = Integer.parseInt(jobInstanceId);
instArray[jobInstNum - 1] = jobInst;
}
Map<JobInstance, List<Integer>> result = new LinkedHashMap<>(jobInstances.size(), 1);
for (int i = 0; i < shardingTotalCount; i++) {
//looup same number JobInstance
JobInstance locatedJobInst = instArray[i];
log.debug("master of {} is {}", i, locatedJobInst);
if (locatedJobInst == null) {
//lookup backup JobInstance
if (backupMap[i] - 1 >= instArray.length || instArray[backupMap[i] - 1] == null) {
throw new RuntimeException("No main or backup job executor found for unit:" + (i + 1) + ", backup unit:" + backupMap[i]);
}
locatedJobInst = instArray[backupMap[i] - 1];
log.debug("backup of {} is {}", i, locatedJobInst);
}
List<Integer> parts = result.get(locatedJobInst);
if (parts == null) {
parts = new ArrayList<>(2);
result.put(locatedJobInst, parts);
}
parts.add(i);
}
return result;
}
@Override
public String getType() {
return "UNIT_ALLOCATION";
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customization the JobInstance id
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-726568398
Detail reason for we need this feathure:
We have a large system that has large data, so we have to use unitized archecture.
So one job instance in specific unit can only execute specific sharding data.
e.g.
JobInstance-unit1 can only handle data sharding No.1
JobInstance-unit2 can only handle data sharding No.2
and so on
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727569831
> jobParameter
@terrymanu
jobParameter can only identify sub-job, NOT jobInstance or job executor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727569644
> @johnny2002 Hi, thank you for the suggestion.
> I want discuss further.
> Job instance id is internal impl in elastic job which user should not focus on them.
> It is not a good idea to expose lots of internal impl to special reason.
>
> Can you consider about reuse the job sharding strategy to rebalance your sub-job?
Hi Terry,
My problem is cannot identify JobInstnaces.
If you don't like to change the jobinstance id, I have another graceful way like:
Pass all System Properties that name starts with "elasticjob." into JobInstance.
Sample code:
```java
public final class JobInstance {
public static final String DELIMITER = "@-@";
private final String jobInstanceId;
private final Map<String, String> properties = new LinkedHashMap<>();
public JobInstance() {
jobInstanceId = IpUtils.getIp() + DELIMITER + ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
//extract all system properties that name start with "elasticjob."
for (Map.Entry propEntry : System.getProperties().entrySet()) {
if (propEntry.getKey().toString().startsWith("elasticjob.")) {
properties.put((String)propEntry.getKey(), (String)propEntry.getValue());
}
}
}
```
Then we can identify JobInstance through JobInstance.properties
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727570089
> > Detail reason for we need this feathure:
> > We have a large system that has large data, so we have to use unitized architecture.
> > So one job instance in specific unit can only execute specific sharding data.
> > e.g.
> > JobInstance-unit1 can only handle data sharding No.1
> > JobInstance-unit2 can only handle data sharding No.2
> > and so on
>
> As you said that your system is partition architecture, why not treat them as different jobs?
Then there will be 1000 jobs need to config one by one. and also hard to monitor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727673999
> > > > Detail reason for we need this feathure:
> > > > We have a large system that has large data, so we have to use unitized architecture.
> > > > So one job instance in specific unit can only execute specific sharding data.
> > > > e.g.
> > > > JobInstance-unit1 can only handle data sharding No.1
> > > > JobInstance-unit2 can only handle data sharding No.2
> > > > and so on
> > >
> > >
> > > As you said that your system is partition architecture, why not treat them as different jobs?
> >
> >
> > Then there will be 1000 jobs need to config one by one. and also hard to monitor.
>
> I don't quite understand your scenarios. Maybe you can just configure different job names while keeping other configuration consistent. I think configuring 1000 job names is as simple as configuring 1000 job instance id. Coorperating with Tracing module, you can immediately locate the problem by job name if any partition failed.
32 job names by 32 instance names, that's 1024 Job Instances, compare to configure 1024 job names, do you think which is simpler?
Another protential problem is that if we seperate one job to many jobs, then the configuration may be inconsistent.
Even though We can configuring 1000 job names, we still need to distinguish main and backup job instance.
Our complete sharding strategy is that:
Firstly, assign sub-job1 to instance1(main), if job instance1 crashed, then assign to instance2(backup)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-728025743
What I mean is we need to design a whole way to provide a project, not for patch only.
Customized job instance id and sharding strategy together is not good for common user.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 commented on issue #1720: Customization the JobInstance id
Posted by GitBox <gi...@apache.org>.
johnny2002 commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727560738
> @johnny2002 Hi, thank you for the suggestion.
> I want discuss further.
> Job instance id is internal impl in elastic job which user should not focus on them.
> It is not a good idea to expose lots of internal impl to special reason.
>
> Can you consider about reuse the job sharding strategy to rebalance your sub-job?
H Terry,
Of course I want to use my own sharding strategy to assign sub-job, while, the job strategy api passed in a list of instances, I need to exactly identify the jobinstance, so that I can assign corresponding sub-job to correct jobInstance. **Each jobinstance only has the resource to handle one specific sub-job.** e.g. JobInstance from unit 1 can only handle sub-job1 or sub-job2, CANNOT handle other sub-jobs.
Sine JobInstance is expose in JobShardingStrategy api, I don't think it is an internal impl.
My sharding strategy is:
sub-job1 assign to jobInstance 1(or you can say Jobexecutor 1)
sub-job2 assign to jobInstance 2
... ...
in JobShardingStrategy api, I need to identify each jobInstances.
```java
public Map<JobInstance, List<Integer>> sharding(final List<JobInstance> jobInstances, final String jobName, final int shardingTotalCount) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727569831
> Can you use jobParameter to recognize your customized sharding logic?
@terrymanu
jobParameter can only identify sub-job, NOT jobInstance or job executor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727560738
> @johnny2002 Hi, thank you for the suggestion.
> I want discuss further.
> Job instance id is internal impl in elastic job which user should not focus on them.
> It is not a good idea to expose lots of internal impl to special reason.
>
> Can you consider about reuse the job sharding strategy to rebalance your sub-job?
H Terry,
Of course I want to use my own sharding strategy to assign sub-job, while, the job strategy api passed in a list of instances, I need to exactly identify the jobinstance, so that I can assign corresponding sub-job to correct jobInstance. In my application, JobInstances are not equivalent. **Each jobinstance only has the resource to handle 1 or 2 specific sub-jobs.** e.g. JobInstance from unit 1 can only handle sub-job1 or sub-job2, CANNOT handle other sub-jobs.
Sine JobInstance is expose in JobShardingStrategy api, I don't think it is an internal impl.
My sharding strategy is:
sub-job1 assign to jobInstance 1(or you can say Jobexecutor 1)
sub-job2 assign to jobInstance 2
...
in JobShardingStrategy api, I need to identify each jobInstances due to they arenot equivalent.
```java
public Map<JobInstance, List<Integer>> sharding(final List<JobInstance> jobInstances, final String jobName, final int shardingTotalCount) {
```
My problem is that in my JobShardingStrategy impl, I have no way to know which JobInstance from which unit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] TeslaCN commented on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727567379
> Detail reason for we need this feathure:
>
> We have a large system that has large data, so we have to use unitized architecture.
> So one job instance in specific unit can only execute specific sharding data.
> e.g.
> JobInstance-unit1 can only handle data sharding No.1
> JobInstance-unit2 can only handle data sharding No.2
> and so on
As you said that your system are partition architecture, why not treat them as different jobs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] terrymanu edited a comment on issue #1720: Customization the JobInstance id
Posted by GitBox <gi...@apache.org>.
terrymanu edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727515701
@johnny2002 Hi, thank you for the suggestion.
I want discuss further.
Job instance id is internal impl in elastic job which user should not focus on them.
It is not a good idea to expose lots of internal impl to special reason.
Can you consider about reuse the job sharding strategy to rebalance your sub-job?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] johnny2002 edited a comment on issue #1720: Customize the JobInstance
Posted by GitBox <gi...@apache.org>.
johnny2002 edited a comment on issue #1720:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/1720#issuecomment-727569644
> @johnny2002 Hi, thank you for the suggestion.
> I want discuss further.
> Job instance id is internal impl in elastic job which user should not focus on them.
> It is not a good idea to expose lots of internal impl to special reason.
>
> Can you consider about reuse the job sharding strategy to rebalance your sub-job?
Hi Terry,
My problem is cannot identify JobInstnaces. **Update on 2020-11-16: Sorry, I found this design not work, please ignore it.**
If you don't like to change the jobinstance id, I have another graceful way like:
Pass all System Properties that name starts with "elasticjob." into JobInstance.
Sample code:
```java
public final class JobInstance {
public static final String DELIMITER = "@-@";
private final String jobInstanceId;
private final Map<String, String> properties = new LinkedHashMap<>();
public JobInstance() {
jobInstanceId = IpUtils.getIp() + DELIMITER + ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
//extract all system properties that name start with "elasticjob."
for (Map.Entry propEntry : System.getProperties().entrySet()) {
if (propEntry.getKey().toString().startsWith("elasticjob.")) {
properties.put((String)propEntry.getKey(), (String)propEntry.getValue());
}
}
}
```
Then we can identify JobInstance through JobInstance.properties
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org