You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2022/09/25 23:01:27 UTC
[GitHub] [systemds] Baunsgaard commented on a diff in pull request #1698: [MINOR] Federated monitoring minor bug fixes and documentation
Baunsgaard commented on code in PR #1698:
URL: https://github.com/apache/systemds/pull/1698#discussion_r979468976
##########
docs/site/federated-monitoring.md:
##########
@@ -0,0 +1,153 @@
+---
+layout: site
+title: Use SystemDS Federated Monitoring Software
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## SystemDS Federated Monitoring Software
+
+### Introduction
+
+To monitor the federated infrastructure of SystemDS, a monitoring tool was developed for this purpose.
+A general overview of the architecture can be seen in [**Figure 1**](figure-1).
+The monitoring tool consists of two separate decoupled modules, the Java-based **monitoring backend** and
+the **monitoring frontend** developed in [Angular](https://angular.io/).
+
+**NOTE:** To work with the monitoring tool both the back- and frontend services must be running!
+
+
+
+### Installation & Build
+
+#### 1. Monitoring Backend
+
+The required packages for SystemDS, can be found [here](./install.md).
+
+To install and build the packages for the backend, the usual process for building and installing
+SystemDS is sufficient:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
Review Comment:
not a generic path for systemds. we need to reformulate this.
##########
src/test/java/org/apache/sysds/test/functions/federated/monitoring/FederatedWorkerStatisticsTest.java:
##########
@@ -58,6 +87,54 @@ public void testWorkerStatisticsParsedCorrectly() {
Assert.assertNotEquals("Utilization stats parsed correctly", 0, model.utilization.size());
}
+ @Test
+ public void testWorkerStatisticsPerformance() throws InterruptedException {
+ ExecutorService executor = Executors.newFixedThreadPool(workerPorts.length);
+
+ double meanExecTime = 0.f;
+ double numRepetitionsExperiment = 100.f;
+
+ for (int j = -10; j < numRepetitionsExperiment; j++) {
+
+ Collection<Callable<StatisticsModel>> collect = new ArrayList<>();
+ Collection<Callable<Boolean>> parse = new ArrayList<>();
+
+ for (int i = 1; i <= workerPorts.length; i++) {
+ long id = i;
+ String address = "localhost:" + workerPorts[i - 1];
+ workerMonitoringService.create(new WorkerModel(id, "Worker", address));
+ collect.add(() -> StatisticsService.getWorkerStatistics(id, address));
+ }
+
+ long start = System.currentTimeMillis();
+
+ // Returns a list of Futures holding their status and results when all complete.
+ // Future.isDone() is true for each element of the returned list
+ // https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorService.html#invokeAll(java.util.Collection)
+ List<Future<StatisticsModel>> taskFutures = executor.invokeAll(collect);
+
+ taskFutures.forEach(res -> parse.add(() -> syncWorkerStats(res.get(), res.get().traffic.get(0).workerId)));
+
+ executor.invokeAll(parse);
+
+ long finish = System.currentTimeMillis();
+ long elapsedTime = (finish - start);
+
+ if (j >= 0) {
+ meanExecTime += elapsedTime;
+ }
+ }
+
+ executor.shutdown();
+
+ // Wait until all threads are finish
+ // Returns true if all tasks have completed following shut down.
+ // Note that isTerminated is never true unless either shutdown or shutdownNow was called first.
+ while (!executor.isTerminated());
+
+ System.out.println(String.format(PERFORMANCE_FORMAT, workerPorts.length, Math.round(meanExecTime / numRepetitionsExperiment)));
Review Comment:
also if possible use Logging framework instead of System.out.println.
https://github.com/apache/systemds/blob/a3ea6c13e89a26e319038294e585cef27365606c/src/main/java/org/apache/sysds/api/DMLScript.java#L155
##########
docs/site/federated-monitoring.md:
##########
@@ -0,0 +1,153 @@
+---
+layout: site
+title: Use SystemDS Federated Monitoring Software
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## SystemDS Federated Monitoring Software
+
+### Introduction
+
+To monitor the federated infrastructure of SystemDS, a monitoring tool was developed for this purpose.
+A general overview of the architecture can be seen in [**Figure 1**](figure-1).
+The monitoring tool consists of two separate decoupled modules, the Java-based **monitoring backend** and
+the **monitoring frontend** developed in [Angular](https://angular.io/).
+
+**NOTE:** To work with the monitoring tool both the back- and frontend services must be running!
+
+
+
+### Installation & Build
+
+#### 1. Monitoring Backend
+
+The required packages for SystemDS, can be found [here](./install.md).
+
+To install and build the packages for the backend, the usual process for building and installing
+SystemDS is sufficient:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Compile SystemDS
+mvn clean package
+```
+
+#### 2. Monitoring Frontend
+
+Since the frontend is in **Angular v13**, a **node version 12/14/16** or later minor version is required.
+
+Example **node and npm** installation for **Ubuntu 20.04**:
+```bash
+# Installation tutorial:
+# https://linuxize.com/post/how-to-install-node-js-on-ubuntu-20-04/
+
+# 1. Download and execute the NodeSource installation script
+curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
+# 2. Install Node.js and npm
+sudo apt install nodejs
Review Comment:
if we can avoid it (and we can ) we should not install the dependency as sudo.
##########
src/main/java/org/apache/sysds/runtime/controlprogram/federated/monitoring/repositories/DerbyRepository.java:
##########
@@ -127,7 +124,7 @@ public <T extends BaseModel> Long createEntity(T model) {
PreparedStatement st = null;
long id = -1L;
- try {
+ try (var db = DriverManager.getConnection(DB_CONNECTION)) {
Review Comment:
this is new to me,
try-with-resources Statement
i had to look this up and did not know such a trick, but very nice. I will use in the future.
##########
src/main/java/org/apache/sysds/conf/DMLConfig.java:
##########
@@ -122,6 +122,7 @@
public static final String FEDERATED_PAR_INST = "sysds.federated.par_inst";
public static final String FEDERATED_PAR_CONN = "sysds.federated.par_conn";
public static final String PRIVACY_CONSTRAINT_MOCK = "sysds.federated.priv_mock";
+ public static final String FEDERATED_MONITOR_FREQUENCY = "sysds.federated.monitorFreq";
Review Comment:
i think we need a comment here saying it is heartbeat frequency. and it is in seconds.
##########
scripts/monitoring/src/app/utils.ts:
##########
@@ -22,7 +22,7 @@ export class Utils {
return a.x < b.x ? -1 : (a.x > b.x ? 1 : 0);
}
- public static sortStartDate(a, b) {
- return a.startTime < b.startTime ? -1 : (a.startTime > b.startTime ? 1 : 0);
+ public static sortEventsStartDate(a, b) {
+ return a.stages[0].startTime < b.stages[0].startTime ? -1 : (a.stages[0].startTime > b.stages[0].startTime ? 1 : 0);
Review Comment:
minor but, could be done more elegantly with a few more lines getting out the a and b stages.
I only comment because the previous looked clean, maybe call the previous in the new ?
##########
docs/site/federated-monitoring.md:
##########
@@ -0,0 +1,153 @@
+---
+layout: site
+title: Use SystemDS Federated Monitoring Software
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## SystemDS Federated Monitoring Software
+
+### Introduction
+
+To monitor the federated infrastructure of SystemDS, a monitoring tool was developed for this purpose.
+A general overview of the architecture can be seen in [**Figure 1**](figure-1).
+The monitoring tool consists of two separate decoupled modules, the Java-based **monitoring backend** and
+the **monitoring frontend** developed in [Angular](https://angular.io/).
+
+**NOTE:** To work with the monitoring tool both the back- and frontend services must be running!
+
+
+
+### Installation & Build
+
+#### 1. Monitoring Backend
+
+The required packages for SystemDS, can be found [here](./install.md).
+
+To install and build the packages for the backend, the usual process for building and installing
+SystemDS is sufficient:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Compile SystemDS
+mvn clean package
+```
+
+#### 2. Monitoring Frontend
+
+Since the frontend is in **Angular v13**, a **node version 12/14/16** or later minor version is required.
+
+Example **node and npm** installation for **Ubuntu 20.04**:
+```bash
+# Installation tutorial:
+# https://linuxize.com/post/how-to-install-node-js-on-ubuntu-20-04/
+
+# 1. Download and execute the NodeSource installation script
+curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
+# 2. Install Node.js and npm
+sudo apt install nodejs
+
+# 3. Verify installation ----
+node --version
+# Output
+# v14.2.0
+
+npm --version
+# Output
+# 6.14.4
+# ---------------------------
+```
+
+To install the npm packages required for the Angular app to run:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go into the directory with the frontend app
+cd scripts/monitoring
+# 3. Install all npm packages
+npm install
+```
+After those steps all the packages needed for running the monitoring tool should be installed.
+
+### Running
+
+Both back- and frontend applications are separate modules of the same tool, they can be independently started and stopped.
+Since they are designed with loose decoupling in mind, the frontend can integrate with different backends, and
+the backend can work with different frontends, provided that the format of the data and the communication protocol is
+preserved.
+
+#### 1. Monitoring Backend
+
+To run the backend, use the `-fedMonitoring` flag followed by a `port`:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go to the target directory
+cd target
+# 3. Start the backend with the -fedMonitoring flag and a port
+java -cp ./lib/*:./SystemDS.jar org.apache.sysds.api.DMLScript -fedMonitoring 8080
+
+# You should see something like this
+#[ INFO] Setting up Federated Monitoring Backend on port 8080
+#[ INFO] Starting Federated Monitoring Backend server at port: 8080
+#[ INFO] Started Federated Monitoring Backend at port: 8080
+```
+This will start the backend server which will be listening for REST requests on `http://localhost:8080`.
+
+**NOTE:** The backend is polling all registered workers with a given frequency, it can be changed by including
+the `<sysds.federated.monitorFreq>3</sysds.federated.monitorFreq>` in the `SystemDS-config.xml` file, accepting
+**doubles**, representing seconds (0.5 can be used for setting the frequency to be half a second). The example shown
+here will start the backend with polling with frequency of **3 seconds**, which is also the default value.
+
+#### 2. Monitoring Frontend
+
+To run the Angular app:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go into the directory with the frontend app
+cd scripts/monitoring
+# 3. Start the angular app
+npm start
+```
+After this step the Angular UI should be started on [http://localhost:4200](http://localhost:4200) and can be viewed by opening the
+browser on the same address.
+
+**NOTE:** The address of the backend is hardcoded in the frontend application and can be changed by changing the `BASE_URI` in the `systemds/scripts/monitoring/app/constants.ts` file. **DO NOT** include a trailing slash `/`, at the end of the address.
+
+#### 3. Coordinator self-registration for monitoring
+
+In addition to the manual registration of coordinators for monitoring, the self-registration feature can be used by
+setting the `-fedMonitoringAddress` flag followed by the address of the backend:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go to the target directory
+cd target
+# 3. Start the coordinator process with the -fedMonitoringAddress flag and the address of the backend
+java -cp ./lib/*:./SystemDS.jar org.apache.sysds.api.DMLScript \
Review Comment:
replace with the systemds bin file.
##########
docs/site/federated-monitoring.md:
##########
@@ -0,0 +1,153 @@
+---
+layout: site
+title: Use SystemDS Federated Monitoring Software
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## SystemDS Federated Monitoring Software
+
+### Introduction
+
+To monitor the federated infrastructure of SystemDS, a monitoring tool was developed for this purpose.
+A general overview of the architecture can be seen in [**Figure 1**](figure-1).
+The monitoring tool consists of two separate decoupled modules, the Java-based **monitoring backend** and
+the **monitoring frontend** developed in [Angular](https://angular.io/).
+
+**NOTE:** To work with the monitoring tool both the back- and frontend services must be running!
+
+
+
+### Installation & Build
+
+#### 1. Monitoring Backend
+
+The required packages for SystemDS, can be found [here](./install.md).
+
+To install and build the packages for the backend, the usual process for building and installing
+SystemDS is sufficient:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Compile SystemDS
+mvn clean package
+```
+
+#### 2. Monitoring Frontend
+
+Since the frontend is in **Angular v13**, a **node version 12/14/16** or later minor version is required.
+
+Example **node and npm** installation for **Ubuntu 20.04**:
+```bash
+# Installation tutorial:
+# https://linuxize.com/post/how-to-install-node-js-on-ubuntu-20-04/
+
+# 1. Download and execute the NodeSource installation script
+curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
+# 2. Install Node.js and npm
+sudo apt install nodejs
+
+# 3. Verify installation ----
+node --version
+# Output
+# v14.2.0
+
+npm --version
+# Output
+# 6.14.4
+# ---------------------------
+```
+
+To install the npm packages required for the Angular app to run:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go into the directory with the frontend app
+cd scripts/monitoring
+# 3. Install all npm packages
+npm install
+```
+After those steps all the packages needed for running the monitoring tool should be installed.
+
+### Running
+
+Both back- and frontend applications are separate modules of the same tool, they can be independently started and stopped.
+Since they are designed with loose decoupling in mind, the frontend can integrate with different backends, and
+the backend can work with different frontends, provided that the format of the data and the communication protocol is
+preserved.
+
+#### 1. Monitoring Backend
+
+To run the backend, use the `-fedMonitoring` flag followed by a `port`:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go to the target directory
+cd target
+# 3. Start the backend with the -fedMonitoring flag and a port
+java -cp ./lib/*:./SystemDS.jar org.apache.sysds.api.DMLScript -fedMonitoring 8080
+
+# You should see something like this
+#[ INFO] Setting up Federated Monitoring Backend on port 8080
+#[ INFO] Starting Federated Monitoring Backend server at port: 8080
+#[ INFO] Started Federated Monitoring Backend at port: 8080
+```
+This will start the backend server which will be listening for REST requests on `http://localhost:8080`.
+
+**NOTE:** The backend is polling all registered workers with a given frequency, it can be changed by including
+the `<sysds.federated.monitorFreq>3</sysds.federated.monitorFreq>` in the `SystemDS-config.xml` file, accepting
+**doubles**, representing seconds (0.5 can be used for setting the frequency to be half a second). The example shown
+here will start the backend with polling with frequency of **3 seconds**, which is also the default value.
+
+#### 2. Monitoring Frontend
+
+To run the Angular app:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
Review Comment:
remove
##########
docs/site/federated-monitoring.md:
##########
@@ -0,0 +1,153 @@
+---
+layout: site
+title: Use SystemDS Federated Monitoring Software
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## SystemDS Federated Monitoring Software
+
+### Introduction
+
+To monitor the federated infrastructure of SystemDS, a monitoring tool was developed for this purpose.
+A general overview of the architecture can be seen in [**Figure 1**](figure-1).
+The monitoring tool consists of two separate decoupled modules, the Java-based **monitoring backend** and
+the **monitoring frontend** developed in [Angular](https://angular.io/).
+
+**NOTE:** To work with the monitoring tool both the back- and frontend services must be running!
+
+
+
+### Installation & Build
+
+#### 1. Monitoring Backend
+
+The required packages for SystemDS, can be found [here](./install.md).
+
+To install and build the packages for the backend, the usual process for building and installing
+SystemDS is sufficient:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Compile SystemDS
+mvn clean package
+```
+
+#### 2. Monitoring Frontend
+
+Since the frontend is in **Angular v13**, a **node version 12/14/16** or later minor version is required.
+
+Example **node and npm** installation for **Ubuntu 20.04**:
+```bash
+# Installation tutorial:
+# https://linuxize.com/post/how-to-install-node-js-on-ubuntu-20-04/
+
+# 1. Download and execute the NodeSource installation script
+curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
+# 2. Install Node.js and npm
+sudo apt install nodejs
+
+# 3. Verify installation ----
+node --version
+# Output
+# v14.2.0
+
+npm --version
+# Output
+# 6.14.4
+# ---------------------------
+```
+
+To install the npm packages required for the Angular app to run:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go into the directory with the frontend app
+cd scripts/monitoring
+# 3. Install all npm packages
+npm install
+```
+After those steps all the packages needed for running the monitoring tool should be installed.
+
+### Running
+
+Both back- and frontend applications are separate modules of the same tool, they can be independently started and stopped.
+Since they are designed with loose decoupling in mind, the frontend can integrate with different backends, and
+the backend can work with different frontends, provided that the format of the data and the communication protocol is
+preserved.
+
+#### 1. Monitoring Backend
+
+To run the backend, use the `-fedMonitoring` flag followed by a `port`:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go to the target directory
+cd target
+# 3. Start the backend with the -fedMonitoring flag and a port
+java -cp ./lib/*:./SystemDS.jar org.apache.sysds.api.DMLScript -fedMonitoring 8080
+
+# You should see something like this
+#[ INFO] Setting up Federated Monitoring Backend on port 8080
+#[ INFO] Starting Federated Monitoring Backend server at port: 8080
+#[ INFO] Started Federated Monitoring Backend at port: 8080
+```
+This will start the backend server which will be listening for REST requests on `http://localhost:8080`.
+
+**NOTE:** The backend is polling all registered workers with a given frequency, it can be changed by including
+the `<sysds.federated.monitorFreq>3</sysds.federated.monitorFreq>` in the `SystemDS-config.xml` file, accepting
+**doubles**, representing seconds (0.5 can be used for setting the frequency to be half a second). The example shown
+here will start the backend with polling with frequency of **3 seconds**, which is also the default value.
+
+#### 2. Monitoring Frontend
+
+To run the Angular app:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go into the directory with the frontend app
+cd scripts/monitoring
+# 3. Start the angular app
+npm start
+```
+After this step the Angular UI should be started on [http://localhost:4200](http://localhost:4200) and can be viewed by opening the
+browser on the same address.
+
+**NOTE:** The address of the backend is hardcoded in the frontend application and can be changed by changing the `BASE_URI` in the `systemds/scripts/monitoring/app/constants.ts` file. **DO NOT** include a trailing slash `/`, at the end of the address.
+
+#### 3. Coordinator self-registration for monitoring
+
+In addition to the manual registration of coordinators for monitoring, the self-registration feature can be used by
+setting the `-fedMonitoringAddress` flag followed by the address of the backend:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go to the target directory
+cd target
+# 3. Start the coordinator process with the -fedMonitoringAddress flag and the address of the backend
+java -cp ./lib/*:./SystemDS.jar org.apache.sysds.api.DMLScript \
+-f testFederated.dml -exec singlenode -explain -debug -stats 20 -fedMonitoringAddress http://localhost:8080
+```
+
+**NOTE:** The backend service should already be running, otherwise the coordinator will not start.
Review Comment:
a new line in the end of the file would be nice.
##########
docs/site/federated-monitoring.md:
##########
@@ -0,0 +1,153 @@
+---
+layout: site
+title: Use SystemDS Federated Monitoring Software
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## SystemDS Federated Monitoring Software
+
+### Introduction
+
+To monitor the federated infrastructure of SystemDS, a monitoring tool was developed for this purpose.
+A general overview of the architecture can be seen in [**Figure 1**](figure-1).
+The monitoring tool consists of two separate decoupled modules, the Java-based **monitoring backend** and
+the **monitoring frontend** developed in [Angular](https://angular.io/).
+
+**NOTE:** To work with the monitoring tool both the back- and frontend services must be running!
+
+
+
+### Installation & Build
+
+#### 1. Monitoring Backend
+
+The required packages for SystemDS, can be found [here](./install.md).
+
+To install and build the packages for the backend, the usual process for building and installing
+SystemDS is sufficient:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Compile SystemDS
+mvn clean package
+```
+
+#### 2. Monitoring Frontend
+
+Since the frontend is in **Angular v13**, a **node version 12/14/16** or later minor version is required.
+
+Example **node and npm** installation for **Ubuntu 20.04**:
+```bash
+# Installation tutorial:
+# https://linuxize.com/post/how-to-install-node-js-on-ubuntu-20-04/
+
+# 1. Download and execute the NodeSource installation script
+curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
+# 2. Install Node.js and npm
+sudo apt install nodejs
+
+# 3. Verify installation ----
+node --version
+# Output
+# v14.2.0
+
+npm --version
+# Output
+# 6.14.4
+# ---------------------------
+```
+
+To install the npm packages required for the Angular app to run:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go into the directory with the frontend app
+cd scripts/monitoring
+# 3. Install all npm packages
+npm install
+```
+After those steps all the packages needed for running the monitoring tool should be installed.
+
+### Running
+
+Both back- and frontend applications are separate modules of the same tool, they can be independently started and stopped.
+Since they are designed with loose decoupling in mind, the frontend can integrate with different backends, and
+the backend can work with different frontends, provided that the format of the data and the communication protocol is
+preserved.
+
+#### 1. Monitoring Backend
+
+To run the backend, use the `-fedMonitoring` flag followed by a `port`:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
Review Comment:
remove
##########
src/main/java/org/apache/sysds/runtime/controlprogram/federated/monitoring/controllers/CoordinatorController.java:
##########
@@ -42,9 +43,28 @@ public FullHttpResponse create(Request request) {
@Override
public FullHttpResponse update(Request request, Long objectId) {
+ var result = coordinatorService.get(objectId);
+
+ if (result == null) {
+ return Response.notFound(Constants.NOT_FOUND_MSG);
+ }
+
var model = MapperService.getModelFromBody(request, CoordinatorModel.class);
- model.generateMonitoringKey();
+ model.id = objectId;
+ if (model.host == null) {
Review Comment:
you might like the syntax:
```
model.host = model.host == null ? result.host : model.host;
```
##########
docs/site/federated-monitoring.md:
##########
@@ -0,0 +1,153 @@
+---
+layout: site
+title: Use SystemDS Federated Monitoring Software
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## SystemDS Federated Monitoring Software
+
+### Introduction
+
+To monitor the federated infrastructure of SystemDS, a monitoring tool was developed for this purpose.
+A general overview of the architecture can be seen in [**Figure 1**](figure-1).
+The monitoring tool consists of two separate decoupled modules, the Java-based **monitoring backend** and
+the **monitoring frontend** developed in [Angular](https://angular.io/).
+
+**NOTE:** To work with the monitoring tool both the back- and frontend services must be running!
+
+
+
+### Installation & Build
+
+#### 1. Monitoring Backend
+
+The required packages for SystemDS, can be found [here](./install.md).
+
+To install and build the packages for the backend, the usual process for building and installing
+SystemDS is sufficient:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Compile SystemDS
+mvn clean package
+```
+
+#### 2. Monitoring Frontend
+
+Since the frontend is in **Angular v13**, a **node version 12/14/16** or later minor version is required.
+
+Example **node and npm** installation for **Ubuntu 20.04**:
+```bash
+# Installation tutorial:
+# https://linuxize.com/post/how-to-install-node-js-on-ubuntu-20-04/
+
+# 1. Download and execute the NodeSource installation script
+curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
+# 2. Install Node.js and npm
+sudo apt install nodejs
+
+# 3. Verify installation ----
+node --version
+# Output
+# v14.2.0
+
+npm --version
+# Output
+# 6.14.4
+# ---------------------------
+```
+
+To install the npm packages required for the Angular app to run:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
Review Comment:
again this should not be here.
##########
src/test/java/org/apache/sysds/test/functions/federated/monitoring/FederatedBackendPerformanceTest.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.sysds.test.functions.federated.monitoring;
+
+import static java.lang.Thread.sleep;
+
+import org.apache.sysds.runtime.controlprogram.federated.monitoring.models.WorkerModel;
+import org.apache.sysds.test.TestConfiguration;
+import org.apache.sysds.test.TestUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.net.http.HttpResponse;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+public class FederatedBackendPerformanceTest extends FederatedMonitoringTestBase {
+ private final static String TEST_NAME = "FederatedBackendPerformanceTest";
+ private final static String TEST_DIR = "functions/federated/monitoring/";
+ private static final String TEST_CLASS_DIR = TEST_DIR + FederatedBackendPerformanceTest.class.getSimpleName() + "/";
+ private static final String PERFORMANCE_FORMAT = "For %d number of requests, milliseconds elapsed %d.";
+
+ private static int[] workerPort;
+
+ @Override
+ public void setUp() {
+ TestUtils.clearAssertionInformation();
+ addTestConfiguration(TEST_NAME, new TestConfiguration(TEST_CLASS_DIR, TEST_NAME, new String[] {"S"}));
+ startFedMonitoring(null);
+ workerPort = startFedWorkers(1);
+ }
+
+ @Test
+ public void testBackendPerformance() throws InterruptedException {
Review Comment:
hmm, i think we might need to change this, since a performance test is not a usual test that verify functionality,
How long time does the test take ? napkin math says (100 + 10 ) * 500 Milli seconds sleep plus the code giving a long running test in the cloud each time.
Could we design some functional tests instead?
##########
src/main/java/org/apache/sysds/runtime/controlprogram/federated/monitoring/services/StatisticsService.java:
##########
@@ -50,34 +53,59 @@ public class StatisticsService {
private static final IRepository entityRepository = new DerbyRepository();
public StatisticsModel getAll(Long workerId, StatisticsOptions options) {
+ CompletableFuture<Void> utilizationFuture = null;
Review Comment:
the lines in this method gets very long, i would like to see them split in multiple functions.
##########
src/main/java/org/apache/sysds/runtime/controlprogram/federated/monitoring/services/WorkerService.java:
##########
@@ -105,61 +114,80 @@ private static synchronized void updateCachedWorkers(List<WorkerModel> workers,
}
}
- private static Runnable syncWorkerStatisticsWithDB() {
- return () -> {
-
- for(Map.Entry<Long, Pair<String, Boolean>> entry : cachedWorkers.entrySet()) {
- Long id = entry.getKey();
- String address = entry.getValue().getLeft();
+ private static synchronized void startStatsCollectionProcess(int threadCount, double frequencySeconds) {
+ if (executorService == null) {
+ executorService = Executors.newScheduledThreadPool(threadCount);
+ executorService.scheduleAtFixedRate(syncWorkerStatisticsRunnable(), 0, Math.round(frequencySeconds * 1000), TimeUnit.MILLISECONDS);
+ }
+ }
- var stats = StatisticsService.getWorkerStatistics(id, address);
+ public static void syncWorkerStatisticsWithDB(StatisticsModel stats, Long id) {
- if (stats != null) {
+ if (stats != null) {
- cachedWorkers.get(id).setValue(true);
+ cachedWorkers.get(id).setValue(true);
- if (stats.utilization != null) {
- entityRepository.createEntity(stats.utilization.get(0));
- }
- if (stats.traffic != null) {
- for (var trafficEntity: stats.traffic) {
- if (trafficEntity.coordinatorId > 0) {
- entityRepository.createEntity(trafficEntity);
- }
+ if (stats.utilization != null) {
+ CompletableFuture.runAsync(() -> entityRepository.createEntity(stats.utilization.get(0)));
Review Comment:
these seem dangerous. Since they are independent futures, that are not maintained, just running.
do we not need to monitor that they work in variables in the main thread ?
##########
docs/site/federated-monitoring.md:
##########
@@ -0,0 +1,153 @@
+---
+layout: site
+title: Use SystemDS Federated Monitoring Software
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## SystemDS Federated Monitoring Software
+
+### Introduction
+
+To monitor the federated infrastructure of SystemDS, a monitoring tool was developed for this purpose.
+A general overview of the architecture can be seen in [**Figure 1**](figure-1).
+The monitoring tool consists of two separate decoupled modules, the Java-based **monitoring backend** and
+the **monitoring frontend** developed in [Angular](https://angular.io/).
+
+**NOTE:** To work with the monitoring tool both the back- and frontend services must be running!
+
+
+
+### Installation & Build
+
+#### 1. Monitoring Backend
+
+The required packages for SystemDS, can be found [here](./install.md).
+
+To install and build the packages for the backend, the usual process for building and installing
+SystemDS is sufficient:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Compile SystemDS
+mvn clean package
+```
+
+#### 2. Monitoring Frontend
+
+Since the frontend is in **Angular v13**, a **node version 12/14/16** or later minor version is required.
+
+Example **node and npm** installation for **Ubuntu 20.04**:
+```bash
+# Installation tutorial:
+# https://linuxize.com/post/how-to-install-node-js-on-ubuntu-20-04/
+
+# 1. Download and execute the NodeSource installation script
+curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
+# 2. Install Node.js and npm
+sudo apt install nodejs
+
+# 3. Verify installation ----
+node --version
+# Output
+# v14.2.0
+
+npm --version
+# Output
+# 6.14.4
+# ---------------------------
+```
+
+To install the npm packages required for the Angular app to run:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go into the directory with the frontend app
+cd scripts/monitoring
+# 3. Install all npm packages
+npm install
+```
+After those steps all the packages needed for running the monitoring tool should be installed.
+
+### Running
+
+Both back- and frontend applications are separate modules of the same tool, they can be independently started and stopped.
+Since they are designed with loose decoupling in mind, the frontend can integrate with different backends, and
+the backend can work with different frontends, provided that the format of the data and the communication protocol is
+preserved.
+
+#### 1. Monitoring Backend
+
+To run the backend, use the `-fedMonitoring` flag followed by a `port`:
+
+```bash
+# 1. Go into the directory of systemds
+cd /home/ubuntu/systemds
+# 2. Go to the target directory
+cd target
+# 3. Start the backend with the -fedMonitoring flag and a port
+java -cp ./lib/*:./SystemDS.jar org.apache.sysds.api.DMLScript -fedMonitoring 8080
Review Comment:
we should change this to
systemds -fedMonitoring 8080
I think if i understand correctly you already modified the bin/systemds file to do so?
##########
src/test/java/org/apache/sysds/test/functions/federated/monitoring/FederatedWorkerStatisticsTest.java:
##########
@@ -58,6 +87,54 @@ public void testWorkerStatisticsParsedCorrectly() {
Assert.assertNotEquals("Utilization stats parsed correctly", 0, model.utilization.size());
}
+ @Test
+ public void testWorkerStatisticsPerformance() throws InterruptedException {
Review Comment:
here it is another performance test, what i think we should avoid.
##########
scripts/staging/SIMD-double-vectors/systemds:
##########
@@ -167,7 +167,7 @@ Worker Usage: $0 [-r] WORKER [SystemDS.jar] <portnumber> [arguments] [-help]
port : The port to open for the federated worker.
-Federated Monitoring Usage: $0 [-r] FEDMONITOR [SystemDS.jar] <portnumber> [arguments] [-help]
+Federated Monitoring Usage: $0 [-r] FEDMONITORING [SystemDS.jar] <portnumber> [arguments] [-help]
Review Comment:
Do not modify this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org