You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "imbajin (via GitHub)" <gi...@apache.org> on 2023/11/22 05:05:52 UTC
[PR] chore(ci): update common 1.2 & upgrade action version [incubator-hugegraph-computer]
imbajin opened a new pull request, #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286
<!--
Thank you very much for contributing to Apache HugeGraph, we are happy that you want to help us improve it!
Here are some tips for you:
1. If this is your first time, please read the [contributing guidelines](https://github.com/apache/hugegraph/blob/master/CONTRIBUTING.md)
2. If a PR will fix/close a issue, type the message "close xxx" (xxx is the link of related issue) in the content, github will auto link it (Required)
3. Name the PR title in "Google Commit Format", start with "feat | fix | perf | refactor | doc | chore",
such like: "feat(core): support the PageRank algorithm" or "fix: wrong break in the compute loop" (module is optional)
skip it if you are unsure about which is the best component.
4. One PR address one issue, better not to mix up multiple issues.
5. Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]` (or click it directly after published)
-->
## Purpose of the PR
- subtask of https://github.com/apache/incubator-hugegraph-commons/pull/133 (upgrade common to v1.2)
<!--
Please explain more context in this section, clarify why the changes are needed.
For example:
- If you propose a new API, clarify the use case for a new API.
- If you fix a bug, you can clarify why it is a bug, and should associated with an issue.
-->
## Main Changes
<!-- Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. These change logs are helpful for better ant faster reviews.)
For example:
- If you introduce a new feature, please show detailed design here or add the link of design documentation.
- If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
- If there is a discussion in the mailing list, please add the link. -->
## Verifying these changes
<!-- Please pick the proper options below -->
- [ ] Trivial rework / code cleanup without any test coverage. (No Need)
- [ ] Already covered by existing tests, such as *(please modify tests here)*.
- [ ] Need tests and can be verified as follows.
<!-- Please provide more details about verification
For example:
- If you test manually, please provide related screenshot.
-->
## Does this PR potentially affect the following parts?
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] Nope
- [ ] Dependencies (add/update license info) <!-- Don't forget to add/update the info in "LICENSE" & "NOTICE" files (both in root & dist module) -->
- [ ] Modify configurations
- [ ] The public API
- [ ] Other affects (typed here)
## Documentation Status
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `Doc - TODO` <!-- Your PR changes impact docs and you will update later -->
- [ ] `Doc - Done` <!-- Related docs have been already added or updated -->
- [ ] `Doc - No Need` <!-- Your PR changes don't impact/need docs -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418841084
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -114,9 +113,9 @@ public void init(Config config) {
LOG.info("{} register MasterService", this);
this.bsp4Master.masterInitDone(this.masterInfo);
- this.workers = this.bsp4Master.waitWorkersInitDone();
+ List<ContainerInfo> workers = this.bsp4Master.waitWorkersInitDone();
LOG.info("{} waited all workers registered, workers count: {}",
- this, this.workers.size());
+ this, workers.size());
LOG.info("{} MasterService initialized", this);
this.inited = true;
Review Comment:
```suggestion
this.inited = true;
this.failed = true;
```
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -114,9 +113,9 @@ public void init(Config config) {
LOG.info("{} register MasterService", this);
this.bsp4Master.masterInitDone(this.masterInfo);
- this.workers = this.bsp4Master.waitWorkersInitDone();
+ List<ContainerInfo> workers = this.bsp4Master.waitWorkersInitDone();
LOG.info("{} waited all workers registered, workers count: {}",
- this, this.workers.size());
+ this, workers.size());
LOG.info("{} MasterService initialized", this);
this.inited = true;
Review Comment:
```suggestion
this.inited = true;
this.failed = true;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418335295
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -114,9 +113,9 @@ public void init(Config config) {
LOG.info("{} register MasterService", this);
this.bsp4Master.masterInitDone(this.masterInfo);
- this.workers = this.bsp4Master.waitWorkersInitDone();
+ List<ContainerInfo> workers = this.bsp4Master.waitWorkersInitDone();
Review Comment:
Why don't assign value to `this.workers`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418808134
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:
##########
@@ -91,57 +89,68 @@ public WorkerService() {
/**
* Init worker service, create the managers used by worker service.
*/
- public void init(Config config) {
- E.checkArgument(!this.inited, "The %s has been initialized", this);
-
- this.serviceThread = Thread.currentThread();
- this.registerShutdownHook();
-
- this.config = config;
-
- this.workerInfo = new ContainerInfo();
- LOG.info("{} Start to initialize worker", this);
-
- this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ public synchronized void init(Config config) {
+ try {
+ LOG.info("{} Prepare to init WorkerService", this);
+ // TODO: what will happen if init() called by multiple threads?
+ E.checkArgument(!this.inited, "The %s has been initialized", this);
- /*
- * Keep the waitMasterInitDone() called before initManagers(),
- * in order to ensure master init() before worker managers init()
- */
- this.masterInfo = this.bsp4Worker.waitMasterInitDone();
+ this.serviceThread = Thread.currentThread();
+ this.registerShutdownHook();
+ this.config = config;
+ this.workerInfo = new ContainerInfo();
- InetSocketAddress address = this.initManagers(this.masterInfo);
- this.workerInfo.updateAddress(address);
+ LOG.info("{} Start to initialize worker", this);
+ this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ /*
+ * Keep the waitMasterInitDone() called before initManagers(),
+ * in order to ensure master init() before worker managers init()
+ */
+ ContainerInfo masterInfo = this.bsp4Worker.waitMasterInitDone();
+ InetSocketAddress address = this.initManagers(masterInfo);
+ this.workerInfo.updateAddress(address);
+ this.loadComputation();
+
+ LOG.info("{} register WorkerService", this);
+ this.bsp4Worker.workerInitDone();
+ this.connectToWorkers();
+
+ this.computeManager = new ComputeManager(this.workerInfo.id(), this.context,
+ this.managers);
+
+ this.managers.initedAll(this.config);
+ LOG.info("{} WorkerService initialized", this);
+ this.inited = true;
+ } catch (Exception e) {
+ LOG.error("Error while initializing WorkerService", e);
+ // TODO: shall we call close() here?
+ this.close();
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1417348985
##########
computer-test/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerServiceTest.java:
##########
@@ -207,24 +189,21 @@ public void testServiceWith2Workers() throws InterruptedException {
ComputerOptions.ALGORITHM_MESSAGE_CLASS,
DoubleValue.class.getName()
);
- MasterService masterService = new MasterService();
- try {
+ try (MasterService masterService = new MasterService()) {
masterService.init(config);
masterService.execute();
} catch (Throwable e) {
LOG.error("Failed to start master", e);
exceptions[2] = e;
} finally {
- masterService.close();
countDownLatch.countDown();
Review Comment:
We need to close the worker if master failed, otherwise the main thread won't be able to finish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1421886867
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:
##########
@@ -91,57 +89,67 @@ public WorkerService() {
/**
* Init worker service, create the managers used by worker service.
*/
- public void init(Config config) {
- E.checkArgument(!this.inited, "The %s has been initialized", this);
-
- this.serviceThread = Thread.currentThread();
- this.registerShutdownHook();
-
- this.config = config;
-
- this.workerInfo = new ContainerInfo();
- LOG.info("{} Start to initialize worker", this);
-
- this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ public synchronized void init(Config config) {
+ try {
+ LOG.info("{} Prepare to init WorkerService", this);
+ // TODO: what will happen if init() called by multiple threads?
+ E.checkArgument(!this.inited, "The %s has been initialized", this);
- /*
- * Keep the waitMasterInitDone() called before initManagers(),
- * in order to ensure master init() before worker managers init()
- */
- this.masterInfo = this.bsp4Worker.waitMasterInitDone();
+ this.serviceThread = Thread.currentThread();
+ this.registerShutdownHook();
+ this.config = config;
+ this.workerInfo = new ContainerInfo();
- InetSocketAddress address = this.initManagers(this.masterInfo);
- this.workerInfo.updateAddress(address);
+ LOG.info("{} Start to initialize worker", this);
+ this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ /*
+ * Keep the waitMasterInitDone() called before initManagers(),
+ * in order to ensure master init() before worker managers init()
+ */
+ ContainerInfo masterInfo = this.bsp4Worker.waitMasterInitDone();
+ InetSocketAddress address = this.initManagers(masterInfo);
+ this.workerInfo.updateAddress(address);
+ this.loadComputation();
+
+ LOG.info("{} register WorkerService", this);
+ this.bsp4Worker.workerInitDone();
+ this.connectToWorkers();
+
+ this.computeManager = new ComputeManager(this.workerInfo.id(), this.context,
+ this.managers);
+
+ this.managers.initedAll(this.config);
+ LOG.info("{} WorkerService initialized", this);
+ this.inited = true;
+ } catch (Exception e) {
Review Comment:
> don't need to try-catch and just log here
seems without `try-catch` the log won't print if exception throws before it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418808134
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:
##########
@@ -91,57 +89,68 @@ public WorkerService() {
/**
* Init worker service, create the managers used by worker service.
*/
- public void init(Config config) {
- E.checkArgument(!this.inited, "The %s has been initialized", this);
-
- this.serviceThread = Thread.currentThread();
- this.registerShutdownHook();
-
- this.config = config;
-
- this.workerInfo = new ContainerInfo();
- LOG.info("{} Start to initialize worker", this);
-
- this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ public synchronized void init(Config config) {
+ try {
+ LOG.info("{} Prepare to init WorkerService", this);
+ // TODO: what will happen if init() called by multiple threads?
+ E.checkArgument(!this.inited, "The %s has been initialized", this);
- /*
- * Keep the waitMasterInitDone() called before initManagers(),
- * in order to ensure master init() before worker managers init()
- */
- this.masterInfo = this.bsp4Worker.waitMasterInitDone();
+ this.serviceThread = Thread.currentThread();
+ this.registerShutdownHook();
+ this.config = config;
+ this.workerInfo = new ContainerInfo();
- InetSocketAddress address = this.initManagers(this.masterInfo);
- this.workerInfo.updateAddress(address);
+ LOG.info("{} Start to initialize worker", this);
+ this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ /*
+ * Keep the waitMasterInitDone() called before initManagers(),
+ * in order to ensure master init() before worker managers init()
+ */
+ ContainerInfo masterInfo = this.bsp4Worker.waitMasterInitDone();
+ InetSocketAddress address = this.initManagers(masterInfo);
+ this.workerInfo.updateAddress(address);
+ this.loadComputation();
+
+ LOG.info("{} register WorkerService", this);
+ this.bsp4Worker.workerInitDone();
+ this.connectToWorkers();
+
+ this.computeManager = new ComputeManager(this.workerInfo.id(), this.context,
+ this.managers);
+
+ this.managers.initedAll(this.config);
+ LOG.info("{} WorkerService initialized", this);
+ this.inited = true;
+ } catch (Exception e) {
+ LOG.error("Error while initializing WorkerService", e);
+ // TODO: shall we call close() here?
+ this.close();
Review Comment:
There should be no need to call close, we registered the shutdown hook.
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:
##########
@@ -152,29 +161,45 @@ private void registerShutdownHook() {
}
/**
- * Stop the worker service. Stop the managers created in
- * {@link #init(Config)}.
+ * Stop the worker service. Stop the managers created in {@link #init(Config)}.
*/
@Override
public synchronized void close() {
- this.checkInited();
+ // TODO: why checkInited() here, if init throws exception, how to close the resource?
+ //this.checkInited();
Review Comment:
Make sense, also remove `this.checkInited()` on MasterService.close();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418804791
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -114,9 +113,9 @@ public void init(Config config) {
LOG.info("{} register MasterService", this);
this.bsp4Master.masterInitDone(this.masterInfo);
- this.workers = this.bsp4Master.waitWorkersInitDone();
+ List<ContainerInfo> workers = this.bsp4Master.waitWorkersInitDone();
Review Comment:
Oh, it's better to print the workers' information so troubleshooting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418808134
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:
##########
@@ -91,57 +89,68 @@ public WorkerService() {
/**
* Init worker service, create the managers used by worker service.
*/
- public void init(Config config) {
- E.checkArgument(!this.inited, "The %s has been initialized", this);
-
- this.serviceThread = Thread.currentThread();
- this.registerShutdownHook();
-
- this.config = config;
-
- this.workerInfo = new ContainerInfo();
- LOG.info("{} Start to initialize worker", this);
-
- this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ public synchronized void init(Config config) {
+ try {
+ LOG.info("{} Prepare to init WorkerService", this);
+ // TODO: what will happen if init() called by multiple threads?
+ E.checkArgument(!this.inited, "The %s has been initialized", this);
- /*
- * Keep the waitMasterInitDone() called before initManagers(),
- * in order to ensure master init() before worker managers init()
- */
- this.masterInfo = this.bsp4Worker.waitMasterInitDone();
+ this.serviceThread = Thread.currentThread();
+ this.registerShutdownHook();
+ this.config = config;
+ this.workerInfo = new ContainerInfo();
- InetSocketAddress address = this.initManagers(this.masterInfo);
- this.workerInfo.updateAddress(address);
+ LOG.info("{} Start to initialize worker", this);
+ this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ /*
+ * Keep the waitMasterInitDone() called before initManagers(),
+ * in order to ensure master init() before worker managers init()
+ */
+ ContainerInfo masterInfo = this.bsp4Worker.waitMasterInitDone();
+ InetSocketAddress address = this.initManagers(masterInfo);
+ this.workerInfo.updateAddress(address);
+ this.loadComputation();
+
+ LOG.info("{} register WorkerService", this);
+ this.bsp4Worker.workerInitDone();
+ this.connectToWorkers();
+
+ this.computeManager = new ComputeManager(this.workerInfo.id(), this.context,
+ this.managers);
+
+ this.managers.initedAll(this.config);
+ LOG.info("{} WorkerService initialized", this);
+ this.inited = true;
+ } catch (Exception e) {
+ LOG.error("Error while initializing WorkerService", e);
+ // TODO: shall we call close() here?
+ this.close();
Review Comment:
There should be no need to call close, we registered the shutdown hook.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418260375
##########
computer-test/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerServiceTest.java:
##########
@@ -207,24 +189,21 @@ public void testServiceWith2Workers() throws InterruptedException {
ComputerOptions.ALGORITHM_MESSAGE_CLASS,
DoubleValue.class.getName()
);
- MasterService masterService = new MasterService();
- try {
+ try (MasterService masterService = new MasterService()) {
masterService.init(config);
masterService.execute();
} catch (Throwable e) {
LOG.error("Failed to start master", e);
exceptions[2] = e;
} finally {
- masterService.close();
countDownLatch.countDown();
Review Comment:
> We need to close the worker if master failed, otherwise the main thread won't be able to finish.
Seems in this case, master close() in the end of the test? (and how could we close worker here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1420294751
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -114,9 +113,9 @@ public void init(Config config) {
LOG.info("{} register MasterService", this);
this.bsp4Master.masterInitDone(this.masterInfo);
- this.workers = this.bsp4Master.waitWorkersInitDone();
+ List<ContainerInfo> workers = this.bsp4Master.waitWorkersInitDone();
LOG.info("{} waited all workers registered, workers count: {}",
- this, this.workers.size());
+ this, workers.size());
LOG.info("{} MasterService initialized", this);
this.inited = true;
Review Comment:
> @imbajin here
consider modify them carefully & better in a separate PR maybe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] chore(ci): update common 1.2 & upgrade action version [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#issuecomment-1840205628
@coderzc we could replace all the **download** & **start service** script to `docker run` if the image exist like https://github.com/apache/incubator-hugegraph-computer/pull/286/commits/adf4c43100c71456d4a95b9f473efdf0c6c57f86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1421701982
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -114,9 +113,9 @@ public void init(Config config) {
LOG.info("{} register MasterService", this);
this.bsp4Master.masterInitDone(this.masterInfo);
- this.workers = this.bsp4Master.waitWorkersInitDone();
+ List<ContainerInfo> workers = this.bsp4Master.waitWorkersInitDone();
LOG.info("{} waited all workers registered, workers count: {}",
- this, this.workers.size());
+ this, workers.size());
LOG.info("{} MasterService initialized", this);
this.inited = true;
Review Comment:
Sorry, you are right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418350829
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -114,9 +113,9 @@ public void init(Config config) {
LOG.info("{} register MasterService", this);
this.bsp4Master.masterInitDone(this.masterInfo);
- this.workers = this.bsp4Master.waitWorkersInitDone();
+ List<ContainerInfo> workers = this.bsp4Master.waitWorkersInitDone();
Review Comment:
seems not used in other space..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418350829
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -114,9 +113,9 @@ public void init(Config config) {
LOG.info("{} register MasterService", this);
this.bsp4Master.masterInitDone(this.masterInfo);
- this.workers = this.bsp4Master.waitWorkersInitDone();
+ List<ContainerInfo> workers = this.bsp4Master.waitWorkersInitDone();
Review Comment:
seems not used in other space.. (only defined in the global but not used besides here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1420167741
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -114,9 +113,9 @@ public void init(Config config) {
LOG.info("{} register MasterService", this);
this.bsp4Master.masterInitDone(this.masterInfo);
- this.workers = this.bsp4Master.waitWorkersInitDone();
+ List<ContainerInfo> workers = this.bsp4Master.waitWorkersInitDone();
LOG.info("{} waited all workers registered, workers count: {}",
- this, this.workers.size());
+ this, workers.size());
LOG.info("{} MasterService initialized", this);
this.inited = true;
Review Comment:
@imbajin here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "javeme (via GitHub)" <gi...@apache.org>.
javeme commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1421435730
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/master/MasterService.java:
##########
@@ -141,24 +140,36 @@ private void registerShutdownHook() {
}
/**
- * Stop the the master service. Stop the managers created in
- * {@link #init(Config)}.
+ * Stop the master service. Stop the managers created in {@link #init(Config)}.
*/
@Override
public synchronized void close() {
- this.checkInited();
+ // TODO: check the logic of close carefully later
+ //this.checkInited();
if (this.closed) {
LOG.info("{} MasterService had closed before", this);
return;
}
- this.masterComputation.close(new DefaultMasterContext());
+ try {
+ if (this.masterComputation != null) {
+ this.masterComputation.close(new DefaultMasterContext());
+ }
+ } catch (Exception e) {
+ LOG.error("Error occurred while closing masterComputation", e);
Review Comment:
masterComputation => master service
##########
checkstyle.xml:
##########
@@ -35,7 +35,7 @@
<module name="TreeWalker">
<!--检查行长度-->
<module name="LineLength">
- <property name="max" value="100"/>
+ <property name="max" value="105"/>
Review Comment:
prefer don't break the rule
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:
##########
@@ -152,29 +160,45 @@ private void registerShutdownHook() {
}
/**
- * Stop the worker service. Stop the managers created in
- * {@link #init(Config)}.
+ * Stop the worker service. Stop the managers created in {@link #init(Config)}.
*/
@Override
public synchronized void close() {
- this.checkInited();
+ // TODO: why checkInited() here, if init throws exception, how to close the resource?
+ //this.checkInited();
if (this.closed) {
LOG.info("{} WorkerService had closed before", this);
return;
}
- this.computeManager.close();
+ try {
+ this.computeManager.close();
+ } catch (Exception e) {
+ LOG.error("Error when closing computeManager", e);
Review Comment:
computeManager => ComputeManager
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:
##########
@@ -152,29 +160,45 @@ private void registerShutdownHook() {
}
/**
- * Stop the worker service. Stop the managers created in
- * {@link #init(Config)}.
+ * Stop the worker service. Stop the managers created in {@link #init(Config)}.
*/
@Override
public synchronized void close() {
- this.checkInited();
+ // TODO: why checkInited() here, if init throws exception, how to close the resource?
+ //this.checkInited();
if (this.closed) {
LOG.info("{} WorkerService had closed before", this);
return;
}
- this.computeManager.close();
+ try {
+ this.computeManager.close();
Review Comment:
`return if computeManager is null` since checkInited is removed
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:
##########
@@ -91,57 +89,67 @@ public WorkerService() {
/**
* Init worker service, create the managers used by worker service.
*/
- public void init(Config config) {
- E.checkArgument(!this.inited, "The %s has been initialized", this);
-
- this.serviceThread = Thread.currentThread();
- this.registerShutdownHook();
-
- this.config = config;
-
- this.workerInfo = new ContainerInfo();
- LOG.info("{} Start to initialize worker", this);
-
- this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ public synchronized void init(Config config) {
+ try {
+ LOG.info("{} Prepare to init WorkerService", this);
+ // TODO: what will happen if init() called by multiple threads?
+ E.checkArgument(!this.inited, "The %s has been initialized", this);
- /*
- * Keep the waitMasterInitDone() called before initManagers(),
- * in order to ensure master init() before worker managers init()
- */
- this.masterInfo = this.bsp4Worker.waitMasterInitDone();
+ this.serviceThread = Thread.currentThread();
+ this.registerShutdownHook();
+ this.config = config;
+ this.workerInfo = new ContainerInfo();
- InetSocketAddress address = this.initManagers(this.masterInfo);
- this.workerInfo.updateAddress(address);
+ LOG.info("{} Start to initialize worker", this);
+ this.bsp4Worker = new Bsp4Worker(this.config, this.workerInfo);
+ /*
+ * Keep the waitMasterInitDone() called before initManagers(),
+ * in order to ensure master init() before worker managers init()
+ */
+ ContainerInfo masterInfo = this.bsp4Worker.waitMasterInitDone();
+ InetSocketAddress address = this.initManagers(masterInfo);
+ this.workerInfo.updateAddress(address);
+ this.loadComputation();
+
+ LOG.info("{} register WorkerService", this);
+ this.bsp4Worker.workerInitDone();
+ this.connectToWorkers();
+
+ this.computeManager = new ComputeManager(this.workerInfo.id(), this.context,
+ this.managers);
+
+ this.managers.initedAll(this.config);
+ LOG.info("{} WorkerService initialized", this);
+ this.inited = true;
+ } catch (Exception e) {
Review Comment:
don't need to try-catch and just log here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418569975
##########
computer-test/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerServiceTest.java:
##########
@@ -207,24 +189,21 @@ public void testServiceWith2Workers() throws InterruptedException {
ComputerOptions.ALGORITHM_MESSAGE_CLASS,
DoubleValue.class.getName()
);
- MasterService masterService = new MasterService();
- try {
+ try (MasterService masterService = new MasterService()) {
masterService.init(config);
masterService.execute();
} catch (Throwable e) {
LOG.error("Failed to start master", e);
exceptions[2] = e;
} finally {
- masterService.close();
countDownLatch.countDown();
Review Comment:
> We can call workerService.close()
but in this case we can't call it because of the worker service **scope** is not shared/enough (In other case we could do that)
##########
computer-test/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerServiceTest.java:
##########
@@ -207,24 +189,21 @@ public void testServiceWith2Workers() throws InterruptedException {
ComputerOptions.ALGORITHM_MESSAGE_CLASS,
DoubleValue.class.getName()
);
- MasterService masterService = new MasterService();
- try {
+ try (MasterService masterService = new MasterService()) {
masterService.init(config);
masterService.execute();
} catch (Throwable e) {
LOG.error("Failed to start master", e);
exceptions[2] = e;
} finally {
- masterService.close();
countDownLatch.countDown();
Review Comment:
> We can call workerService.close()
but in this case we can't call it because the worker service **scope** is not shared/enough (In other case we could do that)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1421883875
##########
checkstyle.xml:
##########
@@ -35,7 +35,7 @@
<module name="TreeWalker">
<!--检查行长度-->
<module name="LineLength">
- <property name="max" value="100"/>
+ <property name="max" value="105"/>
Review Comment:
Update to 101 & add the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1418336508
##########
computer-test/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerServiceTest.java:
##########
@@ -207,24 +189,21 @@ public void testServiceWith2Workers() throws InterruptedException {
ComputerOptions.ALGORITHM_MESSAGE_CLASS,
DoubleValue.class.getName()
);
- MasterService masterService = new MasterService();
- try {
+ try (MasterService masterService = new MasterService()) {
masterService.init(config);
masterService.execute();
} catch (Throwable e) {
LOG.error("Failed to start master", e);
exceptions[2] = e;
} finally {
- masterService.close();
countDownLatch.countDown();
Review Comment:
> Seems in this case, master close() in the end of the test?
I think so.
> how could we close worker here
We can call workerService.close()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] chore(ci): update common 1.2 & upgrade action version [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#issuecomment-1822201981
> ```
> Error: /home/runner/work/incubator-hugegraph-computer/incubator-hugegraph-computer/computer-k8s-operator/src/main/java/org/apache/hugegraph/computer/k8s/operator/OperatorEntrypoint.java:[39,23] package org.apache.http does not exist
> ```
>
> This exception in CI seems to be caused by common 1.2
Yes, I'll adapt it soon (already marked in the commons PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] chore(ci): update common 1.2 & upgrade action version [incubator-hugegraph-computer]
Posted by "simon824 (via GitHub)" <gi...@apache.org>.
simon824 commented on PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#issuecomment-1822200895
```
Error: /home/runner/work/incubator-hugegraph-computer/incubator-hugegraph-computer/computer-k8s-operator/src/main/java/org/apache/hugegraph/computer/k8s/operator/OperatorEntrypoint.java:[39,23] package org.apache.http does not exist
```
This exception in CI seems to be caused by common 1.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] chore(ci): update common 1.2 & upgrade action version [incubator-hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#issuecomment-1824150848
this PR will be update after toolchain upgrade to v1.2
<img width="863" alt="image" src="https://github.com/apache/incubator-hugegraph-computer/assets/17706099/b4c42edb-1c68-45fa-8afe-83ac473fedba">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1417348985
##########
computer-test/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerServiceTest.java:
##########
@@ -207,24 +189,21 @@ public void testServiceWith2Workers() throws InterruptedException {
ComputerOptions.ALGORITHM_MESSAGE_CLASS,
DoubleValue.class.getName()
);
- MasterService masterService = new MasterService();
- try {
+ try (MasterService masterService = new MasterService()) {
masterService.init(config);
masterService.execute();
} catch (Throwable e) {
LOG.error("Failed to start master", e);
exceptions[2] = e;
} finally {
- masterService.close();
countDownLatch.countDown();
Review Comment:
We need to close the worker, otherwise the main thread won't be able to finish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "simon824 (via GitHub)" <gi...@apache.org>.
simon824 merged PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [PR] refact(core): adaptor for common 1.2 & fix a string of possible CI problem [incubator-hugegraph-computer]
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on code in PR #286:
URL: https://github.com/apache/incubator-hugegraph-computer/pull/286#discussion_r1421702312
##########
computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java:
##########
@@ -152,29 +160,45 @@ private void registerShutdownHook() {
}
/**
- * Stop the worker service. Stop the managers created in
- * {@link #init(Config)}.
+ * Stop the worker service. Stop the managers created in {@link #init(Config)}.
*/
@Override
public synchronized void close() {
- this.checkInited();
+ // TODO: why checkInited() here, if init throws exception, how to close the resource?
+ //this.checkInited();
if (this.closed) {
LOG.info("{} WorkerService had closed before", this);
return;
}
- this.computeManager.close();
+ try {
+ this.computeManager.close();
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org