You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/02/24 22:03:02 UTC
[GitHub] [hadoop-ozone] vivekratnavel opened a new pull request #597: Create
REST API to serve Node information and integrate with UI in Recon.
vivekratnavel opened a new pull request #597: Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597
## What changes were proposed in this pull request?
- Added a REST API to serve Node information in Recon
- Added unit tests to test node endpoint response
- Added Robot test to test the node endpoint response
- Disabled Recon tests in Ozone Secure Robot test setup to avoid test failures. This will be enabled in the future after testing and stabilizing Recon in Secure environments.
- Enabled Hash Router in React to fix this issue https://issues.apache.org/jira/browse/HDDS-3011
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2996
## How was this patch tested?
This patch was tested with unit tests, robot tests and manual tests.
One can manually test the patch by compiling locally and bringing up a cluster via docker by
```
cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
docker-compose up -d
```
Then pointing the browser to localhost:9888
This will bring up datanodes UI in Recon as the default. Scale up the number of datanodes or create keys in ozone via freon and see the change in Recon UI.
```
docker-compose scale datanode=6
ozone freon rk --replicationType=RATIS --numOfVolumes=10 --numOfBuckets=10 --numOfKeys=10 --factor=THREE --numOfThreads=20
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request
#597: Create REST API to serve Node information and integrate with UI in
Recon.
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #597: Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r383576409
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodesResponse.java
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.types;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hadoop.ozone.web.utils.JsonUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+
+/**
+ * Class that represents the API Response structure of Datanodes.
+ */
+public class DatanodesResponse {
+ /**
+ * Total count of the datanodes.
+ */
+ @JsonProperty("totalCount")
+ private long totalCount;
+
+ /**
+ * An array of datanodes.
+ */
+ @JsonProperty("datanodes")
+ private Collection<DatanodeMetadata> datanodes;
+
+
+ public DatanodesResponse() {
+ this(0, new ArrayList<>());
+ }
+
+ public DatanodesResponse(long totalCount,
+ Collection<DatanodeMetadata> datanodes) {
+ this.totalCount = totalCount;
+ this.datanodes = datanodes;
+ }
+
+ public String toJsonString() {
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#597: Create REST API to serve Node information and integrate with UI in
Recon.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #597: Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r383543818
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NodeEndpoint.java
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeState;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.node.states.NodeNotFoundException;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.recon.api.types.DatanodeMetadata;
+import org.apache.hadoop.ozone.recon.api.types.DatanodeStorageReport;
+import org.apache.hadoop.ozone.recon.api.types.DatanodesResponse;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+
+import javax.inject.Inject;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Endpoint to fetch details about datanodes.
+ */
+@Path("/datanodes")
+@Produces(MediaType.APPLICATION_JSON)
+public class NodeEndpoint {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(NodeEndpoint.class);
+
+ private ReconNodeManager nodeManager;
+
+ @Inject
+ NodeEndpoint(OzoneStorageContainerManager reconSCMFacade) {
Review comment:
Nit. Rename 'reconSCMFacade' to 'reconSCM' or 'scm'. Helps with readability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #597: HDDS-2996. Create
REST API to serve Node information and integrate with UI in Recon.
Posted by GitBox <gi...@apache.org>.
elek closed pull request #597: HDDS-2996. Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request
#597: Create REST API to serve Node information and integrate with UI in
Recon.
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #597: Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r383576726
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodesResponse.java
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.types;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hadoop.ozone.web.utils.JsonUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+
+/**
+ * Class that represents the API Response structure of Datanodes.
+ */
+public class DatanodesResponse {
+ /**
+ * Total count of the datanodes.
+ */
+ @JsonProperty("totalCount")
+ private long totalCount;
Review comment:
We would need this "totalCount" as part of DatanodeMetadata response to support server side pagination in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request
#597: HDDS-2996. Create REST API to serve Node information and integrate
with UI in Recon.
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #597: HDDS-2996. Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r384031716
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
##########
@@ -37,7 +37,7 @@ execute_robot_test s3g s3
execute_robot_test scm scmcli
-execute_robot_test scm recon
+# execute_robot_test scm recon
Review comment:
@elek I updated the robot test and docker configs to enable recon as a passive SCM and test the datanodes endpoint. There is still some work needs to be done to support this setup in a secure environment since Datanodes fail to connect to recon and recon fails to connect to SCM with kerberos auth failure. I am planning to handle this in a separate JIRA - https://issues.apache.org/jira/browse/HDDS-3071
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#597: Create REST API to serve Node information and integrate with UI in
Recon.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #597: Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r383547467
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodesResponse.java
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.types;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hadoop.ozone.web.utils.JsonUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+
+/**
+ * Class that represents the API Response structure of Datanodes.
+ */
+public class DatanodesResponse {
+ /**
+ * Total count of the datanodes.
+ */
+ @JsonProperty("totalCount")
+ private long totalCount;
+
+ /**
+ * An array of datanodes.
+ */
+ @JsonProperty("datanodes")
+ private Collection<DatanodeMetadata> datanodes;
+
+
+ public DatanodesResponse() {
+ this(0, new ArrayList<>());
+ }
+
+ public DatanodesResponse(long totalCount,
+ Collection<DatanodeMetadata> datanodes) {
+ this.totalCount = totalCount;
+ this.datanodes = datanodes;
+ }
+
+ public String toJsonString() {
Review comment:
Do we need this method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #597:
HDDS-2996. Create REST API to serve Node information and integrate with UI
in Recon.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #597: HDDS-2996. Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r384329467
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NodeEndpoint.java
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeState;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.node.states.NodeNotFoundException;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.recon.api.types.DatanodeMetadata;
+import org.apache.hadoop.ozone.recon.api.types.DatanodeStorageReport;
+import org.apache.hadoop.ozone.recon.api.types.DatanodesResponse;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+
+import javax.inject.Inject;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Endpoint to fetch details about datanodes.
+ */
+@Path("/datanodes")
+@Produces(MediaType.APPLICATION_JSON)
+public class NodeEndpoint {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(NodeEndpoint.class);
+
+ private ReconNodeManager nodeManager;
+
+ @Inject
+ NodeEndpoint(OzoneStorageContainerManager reconSCM) {
+ this.nodeManager =
+ (ReconNodeManager) reconSCM.getScmNodeManager();
Review comment:
Thanks the answer. I had similar suspicion. IMHO even without dependency injection we can do better separation between creating and wiring the objects on the SCM side. For example we can do the wiring on the SCM side in a separated utility and in this case you can do better wiring on recon.
Or even we can introduce dependency injection on the SCM side as well.
(Not in this patch. Just thinking loudly ;-))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #597:
HDDS-2996. Create REST API to serve Node information and integrate with UI
in Recon.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #597: HDDS-2996. Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r383941533
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NodeEndpoint.java
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeState;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.node.states.NodeNotFoundException;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.recon.api.types.DatanodeMetadata;
+import org.apache.hadoop.ozone.recon.api.types.DatanodeStorageReport;
+import org.apache.hadoop.ozone.recon.api.types.DatanodesResponse;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+
+import javax.inject.Inject;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Endpoint to fetch details about datanodes.
+ */
+@Path("/datanodes")
+@Produces(MediaType.APPLICATION_JSON)
+public class NodeEndpoint {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(NodeEndpoint.class);
+
+ private ReconNodeManager nodeManager;
+
+ @Inject
+ NodeEndpoint(OzoneStorageContainerManager reconSCM) {
+ this.nodeManager =
+ (ReconNodeManager) reconSCM.getScmNodeManager();
Review comment:
Is there any reason to inject `OzoneStorageContainerManager` instead of `ReconNodeManager` (just curious).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on issue #597: HDDS-2996.
Create REST API to serve Node information and integrate with UI in Recon.
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on issue #597: HDDS-2996. Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#issuecomment-591601888
Thanks a lot for the review @avijayanhwx and @elek.
@elek I am going to wait sometime to see how multi-raft works out and then decide how we could improve the display of pipelines in Datanodes page. Potentially there could be too many active pipelines per datanode with multi-raft and I am not sure if it would help the user to list all of them in Datanodes page. But, there is definitely some scope for improvement here. Thanks for your idea!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #597:
HDDS-2996. Create REST API to serve Node information and integrate with UI
in Recon.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #597: HDDS-2996. Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r383917852
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
##########
@@ -37,7 +37,7 @@ execute_robot_test s3g s3
execute_robot_test scm scmcli
-execute_robot_test scm recon
+# execute_robot_test scm recon
Review comment:
Is it intentional. As I see you updated the robot file as well. Does it work. If not (yet) some comments can be added what is missing. Maybe with a JIRA link.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#597: Create REST API to serve Node information and integrate with UI in
Recon.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #597: Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r383545260
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/DatanodesResponse.java
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.types;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hadoop.ozone.web.utils.JsonUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+
+/**
+ * Class that represents the API Response structure of Datanodes.
+ */
+public class DatanodesResponse {
+ /**
+ * Total count of the datanodes.
+ */
+ @JsonProperty("totalCount")
+ private long totalCount;
Review comment:
Can we add a separate endpoint to get number of registered Nodes in Recon? Instead of embedding it inside the DatanodeMetadata response.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #597:
HDDS-2996. Create REST API to serve Node information and integrate with UI
in Recon.
Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #597: HDDS-2996. Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r384331284
##########
File path: hadoop-ozone/dist/src/main/compose/ozonesecure/test.sh
##########
@@ -37,7 +37,7 @@ execute_robot_test s3g s3
execute_robot_test scm scmcli
-execute_robot_test scm recon
+# execute_robot_test scm recon
Review comment:
Thanks a lot. I will add this link to the comment during the commit to make it easier to follow from the code. (Hope you don't mind...)
```
# TODO: https://issues.apache.org/jira/browse/HDDS-3071
# execute_robot_test scm recon
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#597: HDDS-2996. Create REST API to serve Node information and integrate
with UI in Recon.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #597: HDDS-2996. Create REST API to serve Node information and integrate with UI in Recon.
URL: https://github.com/apache/hadoop-ozone/pull/597#discussion_r384121647
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/NodeEndpoint.java
##########
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeState;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.node.states.NodeNotFoundException;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.recon.api.types.DatanodeMetadata;
+import org.apache.hadoop.ozone.recon.api.types.DatanodeStorageReport;
+import org.apache.hadoop.ozone.recon.api.types.DatanodesResponse;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+
+import javax.inject.Inject;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Endpoint to fetch details about datanodes.
+ */
+@Path("/datanodes")
+@Produces(MediaType.APPLICATION_JSON)
+public class NodeEndpoint {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(NodeEndpoint.class);
+
+ private ReconNodeManager nodeManager;
+
+ @Inject
+ NodeEndpoint(OzoneStorageContainerManager reconSCM) {
+ this.nodeManager =
+ (ReconNodeManager) reconSCM.getScmNodeManager();
Review comment:
@elek Thanks for reviewing this. We cannot inject 'ReconNodeManager' since it is not part of the injected classes in the Controller Module. If we bring in the node manager through injection, we will have to move all the other sub modules within Recon SCM also into injection. Currently, they are instantiated very similar to the way in actual SCM, staying true to the "Recon as Passive SCM" story.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org