You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "lxorc (via GitHub)" <gi...@apache.org> on 2023/03/30 16:56:07 UTC
[GitHub] [dolphinscheduler] lxorc opened a new pull request, #13838: fix #13837 the Invalid log : worker worker have not received the hear…
lxorc opened a new pull request, #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838
fix #13837 the Invalid log : worker worker have not received the hearbet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix #13837 the Invalid log : worker worker have not received the hear… [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #13838: fix #13837 the Invalid log : worker worker have not received the hear…
URL: https://github.com/apache/dolphinscheduler/pull/13838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #13838: fix #13837 the Invalid log : worker worker have not received the hear…
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on PR #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838#issuecomment-1619439548
Please fix the code style by `mvn spotless:apply`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #13838: fix #13837 the Invalid log : worker worker have not received the hear…
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on PR #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838#issuecomment-1492804927
> > > > We don't need to add this check, this case might be due to we registering an empty node? Could you please describe how to reproduce this.
> > >
> > >
> > > Yes, by default a `worker` node will be loaded in `registryClient.subscribe(REGISTRY_DOLPHINSCHEDULER_WORKERS, new WorkerDataListener()); `
> > > ```java
> > > registryClient.subscribe(REGISTRY_DOLPHINSCHEDULER_WORKERS, new WorkerDataListener());
> > > String[] parts = path.split("/"); // REGISTRY_DOLPHINSCHEDULER_WORKERS = /nodes/worker
> > > final String workerAddress = parts[parts.length - 1]; // workerAddress = worker
> > > if (type == Type.UPDATE) // the Type.UPDATE is Node /nodes/worker update , case NODE_UPDATED: type(Type.UPDATE);
> > > syncSingleWorkerNodeInfo(workerAddress, JSONUtils.parseObject(data, WorkerHeartBeat.class)); // ???
> > > workerNodeInfo.put(workerAddress, info); // put "worker" key into the workerNodeInfo
> > > ```
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > The effect is that the `addr` of `getHostWeight` will be the `worker` and `heartBeat == null`.
> > > so
> > > ```java
> > > if (heartBeat == null) {
> > > logger.warn("worker {} in work group {} have not received the heartbeat", addr, workerGroup);
> > > return Optional.empty();
> > > }
> > > ```
> >
> >
> > So we only need to subscribe the event of REGISTRY_DOLPHINSCHEDULER_WORKERS's child path
>
> Please help to review my code to see if there are other problems. If there is a better implementation, please reply to me. And I have fixed it in my production environment and it works fine.
Does master path will have this problem? could you please change the master together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #13838: fix #13837 the Invalid log : worker worker have not received the hear…
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on PR #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838#issuecomment-1491183043
We don't need to add this check, this case might be due to we registering an empty node?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #13838: fix #13837 the Invalid log : worker worker have not received the hear…
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on PR #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838#issuecomment-1491246163
> > We don't need to add this check, this case might be due to we registering an empty node? Could you please describe how to reproduce this.
>
> Yes, by default a `worker` node will be loaded in `registryClient.subscribe(REGISTRY_DOLPHINSCHEDULER_WORKERS, new WorkerDataListener()); `
>
> ```java
> registryClient.subscribe(REGISTRY_DOLPHINSCHEDULER_WORKERS, new WorkerDataListener());
> String[] parts = path.split("/"); // REGISTRY_DOLPHINSCHEDULER_WORKERS = /nodes/worker
> final String workerAddress = parts[parts.length - 1]; // workerAddress = worker
> if (type == Type.UPDATE) // the Type.UPDATE is Node /nodes/worker update , case NODE_UPDATED: type(Type.UPDATE);
> syncSingleWorkerNodeInfo(workerAddress, JSONUtils.parseObject(data, WorkerHeartBeat.class)); // ???
> workerNodeInfo.put(workerAddress, info); // put "worker" key into the workerNodeInfo
> ```
>
> The effect is that the `addr` of `getHostWeight` will be the `worker` and `heartBeat == null`.
>
> so
>
> ```java
> if (heartBeat == null) {
> logger.warn("worker {} in work group {} have not received the heartbeat", addr, workerGroup);
> return Optional.empty();
> }
> ```
So we only need to subscribe the child path of REGISTRY_DOLPHINSCHEDULER_WORKERS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lxorc commented on pull request #13838: fix #13837 the Invalid log : worker worker have not received the hear…
Posted by "lxorc (via GitHub)" <gi...@apache.org>.
lxorc commented on PR #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838#issuecomment-1491214786
> We don't need to add this check, this case might be due to we registering an empty node? Could you please describe how to reproduce this.
Yes, by default a `worker` node will be loaded in `registryClient.subscribe(REGISTRY_DOLPHINSCHEDULER_WORKERS, new WorkerDataListener()); `
```java
registryClient.subscribe(REGISTRY_DOLPHINSCHEDULER_WORKERS, new WorkerDataListener());
String[] parts = path.split("/"); // REGISTRY_DOLPHINSCHEDULER_WORKERS = /nodes/worker
final String workerAddress = parts[parts.length - 1]; // workerAddress = worker
syncSingleWorkerNodeInfo(workerAddress, JSONUtils.parseObject(data, WorkerHeartBeat.class));
workerNodeInfo.put(workerAddress, info); // put "worker" key into the workerNodeInfo
` ` `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lxorc commented on pull request #13838: fix #13837 the Invalid log : worker worker have not received the hear…
Posted by "lxorc (via GitHub)" <gi...@apache.org>.
lxorc commented on PR #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838#issuecomment-1491329287
> > > We don't need to add this check, this case might be due to we registering an empty node? Could you please describe how to reproduce this.
> >
> >
> > Yes, by default a `worker` node will be loaded in `registryClient.subscribe(REGISTRY_DOLPHINSCHEDULER_WORKERS, new WorkerDataListener()); `
> > ```java
> > registryClient.subscribe(REGISTRY_DOLPHINSCHEDULER_WORKERS, new WorkerDataListener());
> > String[] parts = path.split("/"); // REGISTRY_DOLPHINSCHEDULER_WORKERS = /nodes/worker
> > final String workerAddress = parts[parts.length - 1]; // workerAddress = worker
> > if (type == Type.UPDATE) // the Type.UPDATE is Node /nodes/worker update , case NODE_UPDATED: type(Type.UPDATE);
> > syncSingleWorkerNodeInfo(workerAddress, JSONUtils.parseObject(data, WorkerHeartBeat.class)); // ???
> > workerNodeInfo.put(workerAddress, info); // put "worker" key into the workerNodeInfo
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > The effect is that the `addr` of `getHostWeight` will be the `worker` and `heartBeat == null`.
> > so
> > ```java
> > if (heartBeat == null) {
> > logger.warn("worker {} in work group {} have not received the heartbeat", addr, workerGroup);
> > return Optional.empty();
> > }
> > ```
>
> So we only need to subscribe the event of REGISTRY_DOLPHINSCHEDULER_WORKERS's child path
Please help to review my code to see if there are other problems. If there is a better implementation, please reply to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix #13837 the Invalid log : worker worker have not received the hear… [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838#issuecomment-1902460330
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Gallardot commented on a diff in pull request #13838: fix #13837 the Invalid log : worker worker have not received the hear…
Posted by "Gallardot (via GitHub)" <gi...@apache.org>.
Gallardot commented on code in PR #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838#discussion_r1308347543
##########
dolphinscheduler-registry/dolphinscheduler-registry-api/src/main/java/org/apache/dolphinscheduler/registry/api/RegistryClient.java:
##########
@@ -210,7 +210,7 @@ public boolean isMasterPath(String path) {
}
public boolean isWorkerPath(String path) {
- return path != null && path.startsWith(RegistryNodeType.WORKER.getRegistryPath());
+ return path != null && path.startsWith(Constants.REGISTRY_DOLPHINSCHEDULER_WORKERS) && !path.equals(Constants.REGISTRY_DOLPHINSCHEDULER_WORKERS);
Review Comment:
```suggestion
return path != null && path.startsWith(Constants.REGISTRY_DOLPHINSCHEDULER_WORKERS)
&& !path.equals(Constants.REGISTRY_DOLPHINSCHEDULER_WORKERS);
```
@lxorc Do you have time to work on this PR? Maybe I can help with that. We also encountered this issue, and I have tested this PR in our environment. It is feasible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix #13837 the Invalid log : worker worker have not received the hear… [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13838:
URL: https://github.com/apache/dolphinscheduler/pull/13838#issuecomment-1890174890
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org