You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "featzhang (via GitHub)" <gi...@apache.org> on 2026/03/01 05:03:33 UTC
[PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
featzhang opened a new pull request, #27716:
URL: https://github.com/apache/flink/pull/27716
## What is the purpose of the change
This PR adds a new **Node Health** tab to the Flink Web UI under the Job Manager page, allowing operators to observe which nodes are currently quarantined by the `NodeHealthManager`.
The page calls the existing `GET /cluster/blocklist` REST API (introduced in PR-3) and renders a table showing:
- **Node ID** – the ResourceID of the quarantined node
- **Cause** – the reason the node was quarantined
- **Expiration Time** – when the quarantine expires (or "Never" for permanent)
- **Status** – a tag indicating `Quarantined` (active) or `Expired`
## Brief change log
- Add `BlockedNodeInfo` and `BlocklistResponse` TypeScript interfaces (`node-health.ts`)
- Add `loadBlocklist()` method to `JobManagerService` calling `GET /cluster/blocklist`
- Add `JobManagerNodeHealthComponent` with an `nz-table` displaying node health status
- Register new route `node-health` under job-manager routes
- Add **Node Health** navigation tab to `JobManagerComponent`
## Verifying this change
1. Start a Flink cluster with `node.health.enabled: true`
2. Use the REST API to quarantine a node:
```
POST /cluster/nodes/{nodeId}/quarantine
{ "reason": "manual test", "duration": "10 min" }
```
3. Open the Flink Web UI → Job Manager → **Node Health** tab
4. Verify the quarantined node appears in the table with correct cause and expiration time
5. After expiration, verify the status tag changes to **Expired**
## Does this pull request potentially affect one of the following parts?
- [ ] Dependencies (does it add or upgrade a dependency): **no**
- [ ] The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
- [ ] The serializers: **no**
- [ ] The runtime per-record code paths (performance sensitive): **no**
- [ ] Anything that affects the Flink WebUI: **yes**
## Documentation
- Does this pull request introduce a new feature? **yes** (Web UI observability for node health)
- If yes, how is the feature documented? Visible in the Web UI; REST API documented in PR-3.
## Depends On
This PR depends on:
- PR-3: `[FLINK-39176][Runtime] Add REST API for Node Quarantine` – provides `GET /cluster/blocklist` endpoint
- PR-1, PR-2, PR-4, PR-5: NodeHealthManager infrastructure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873300120
##########
flink-runtime/src/main/java/org/apache/flink/runtime/management/blocklist/ManagementBlocklistHandler.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.management.blocklist;
+
+import org.apache.flink.runtime.blocklist.BlockedNode;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Set;
+
+/**
+ * Interface for managing cluster-level node blocklist. This is independent of batch execution
+ * blocklist and speculative execution.
+ */
+public interface ManagementBlocklistHandler {
+
+ /**
+ * Add a node to the management blocklist.
+ *
+ * @param nodeId the ID of the node to block
+ * @param reason the reason for blocking the node
+ * @param duration the duration for which the node should be blocked
+ */
+ void addBlockedNode(String nodeId, String reason, Duration duration);
+
+ /**
+ * Remove a node from the management blocklist.
+ *
+ * @param nodeId the ID of the node to unblock
+ * @return true if the node was removed, false if it wasn't in the blocklist
+ */
+ boolean removeBlockedNode(String nodeId);
+
+ /**
+ * Get all currently blocked nodes in the management blocklist.
+ *
+ * @return a set of all blocked nodes
+ */
+ Set<BlockedNode> getAllBlockedNodes();
+
+ /**
+ * Check if a specific node is blocked in the management blocklist.
+ *
+ * @param nodeId the ID of the node to check
+ * @return true if the node is blocked, false otherwise
+ */
+ boolean isNodeBlocked(String nodeId);
+
+ /**
+ * Remove all expired nodes from the management blocklist. This method is called periodically to
Review Comment:
The flip says : "endTimestamp: The end time of the blocking (at which time the node will become available again)." I am curious , is the node available when it expires or when the expired node is removed from the blocked list.
If it is the latter - maybe change the name of this method to makeNodesAvailable().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2872736912
##########
flink-core/src/main/java/org/apache/flink/configuration/ManagementOptions.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.configuration;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.docs.Documentation;
+
+import java.time.Duration;
+
+import static org.apache.flink.configuration.ConfigOptions.key;
+
+/** Configuration options for cluster management features. */
+@PublicEvolving
+public class ManagementOptions {
+
+ @Documentation.Section({Documentation.Sections.EXPERT_SCHEDULING})
+ public static final ConfigOption<Boolean> BLOCKLIST_ENABLED =
+ key("cluster.management.blocklist.enabled")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ "A flag to enable or disable the management blocklist functionality. "
+ + "When enabled, nodes can be manually added to a blocklist via REST API "
Review Comment:
I suggest it is clearer to say taskmanagers rather than nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873341753
##########
docs/content/docs/ops/management_blocklist.md:
##########
@@ -0,0 +1,218 @@
+---
+title: "Management Blocklist"
+weight: 9
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Management Blocklist
+
+The Management Blocklist is a feature that allows cluster administrators to temporarily block problematic nodes from receiving new task allocations. This is useful for maintenance, troubleshooting, or handling nodes with performance issues without completely removing them from the cluster.
+
+## Overview
+
+The Management Blocklist provides a way to:
+- **Block nodes**: Prevent new slots from being allocated on specific nodes
+- **Unblock nodes**: Allow previously blocked nodes to receive new allocations again
+- **View blocked nodes**: List all currently blocked nodes with their block reasons and expiration times
+- **Automatic cleanup**: Blocked nodes are automatically unblocked after a specified duration
+
+Unlike the [speculative execution blocklist]({{< ref "docs/deployment/speculative_execution" >}}), the Management Blocklist is manually controlled by administrators and is intended for operational management rather than automatic performance optimization.
+
+## Configuration
+
+The Management Blocklist can be configured using the following options:
+
+| Configuration Key | Default | Description |
+|-------------------|---------|-------------|
+| `management.blocklist.enabled` | `false` | Whether to enable the management blocklist feature |
+| `management.blocklist.default-duration` | `1h` | Default duration for blocking nodes when no explicit duration is provided |
+| `management.blocklist.max-duration` | `24h` | Maximum allowed duration for blocking nodes |
+
+Example configuration:
+```yaml
+management:
+ blocklist:
+ enabled: true
+ default-duration: 30min
+ max-duration: 12h
+```
+
+## REST API
+
+The Management Blocklist provides a REST API for programmatic control:
+
+### Add Node to Blocklist
+
+**Endpoint**: `POST /cluster/blocklist`
+
+**Request Body**:
+```json
+{
+ "nodeId": "taskmanager-1",
+ "reason": "Maintenance required",
+ "duration": "PT2H"
+}
+```
+
+**Parameters**:
+- `nodeId` (required): The ID of the node to block
+- `reason` (required): Human-readable reason for blocking the node
+- `duration` (optional): Duration to block the node (ISO 8601 format). If not provided, uses the configured default duration.
+
+**Response**:
+```json
+{
+ "status": "success",
Review Comment:
what are the potential errors here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "featzhang (via GitHub)" <gi...@apache.org>.
featzhang commented on PR #27716:
URL: https://github.com/apache/flink/pull/27716#issuecomment-3994920802
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873361207
##########
docs/content/docs/ops/management_blocklist.md:
##########
@@ -0,0 +1,218 @@
+---
+title: "Management Blocklist"
+weight: 9
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Management Blocklist
+
+The Management Blocklist is a feature that allows cluster administrators to temporarily block problematic nodes from receiving new task allocations. This is useful for maintenance, troubleshooting, or handling nodes with performance issues without completely removing them from the cluster.
+
+## Overview
+
+The Management Blocklist provides a way to:
+- **Block nodes**: Prevent new slots from being allocated on specific nodes
+- **Unblock nodes**: Allow previously blocked nodes to receive new allocations again
+- **View blocked nodes**: List all currently blocked nodes with their block reasons and expiration times
+- **Automatic cleanup**: Blocked nodes are automatically unblocked after a specified duration
+
+Unlike the [speculative execution blocklist]({{< ref "docs/deployment/speculative_execution" >}}), the Management Blocklist is manually controlled by administrators and is intended for operational management rather than automatic performance optimization.
+
+## Configuration
+
+The Management Blocklist can be configured using the following options:
+
+| Configuration Key | Default | Description |
+|-------------------|---------|-------------|
+| `management.blocklist.enabled` | `false` | Whether to enable the management blocklist feature |
+| `management.blocklist.default-duration` | `1h` | Default duration for blocking nodes when no explicit duration is provided |
+| `management.blocklist.max-duration` | `24h` | Maximum allowed duration for blocking nodes |
+
+Example configuration:
+```yaml
+management:
+ blocklist:
+ enabled: true
+ default-duration: 30min
+ max-duration: 12h
+```
+
+## REST API
+
+The Management Blocklist provides a REST API for programmatic control:
+
+### Add Node to Blocklist
+
+**Endpoint**: `POST /cluster/blocklist`
+
+**Request Body**:
+```json
+{
+ "nodeId": "taskmanager-1",
+ "reason": "Maintenance required",
+ "duration": "PT2H"
+}
+```
+
+**Parameters**:
+- `nodeId` (required): The ID of the node to block
+- `reason` (required): Human-readable reason for blocking the node
+- `duration` (optional): Duration to block the node (ISO 8601 format). If not provided, uses the configured default duration.
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been added to the blocklist"
+}
+```
+
+### Remove Node from Blocklist
+
+**Endpoint**: `DELETE /cluster/blocklist/{nodeId}`
+
+**Parameters**:
+- `nodeId` (path parameter): The ID of the node to unblock
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been removed from the blocklist"
+}
+```
+
+### List Blocked Nodes
+
+**Endpoint**: `GET /cluster/blocklist`
+
+**Response**:
+```json
+{
+ "blockedNodes": [
+ {
+ "nodeId": "taskmanager-1",
+ "cause": "Maintenance required",
+ "blockedAt": "2024-03-15T10:30:00Z",
+ "expiresAt": "2024-03-15T12:30:00Z"
+ },
+ {
+ "nodeId": "taskmanager-3",
+ "cause": "Performance issues",
+ "blockedAt": "2024-03-15T11:00:00Z",
+ "expiresAt": "2024-03-15T13:00:00Z"
+ }
+ ]
+}
+```
+
+## Usage Examples
+
+### Using curl
+
+```bash
+# Add a node to blocklist for 2 hours
+curl -X POST http://localhost:8081/cluster/blocklist \
+ -H "Content-Type: application/json" \
+ -d '{
+ "nodeId": "taskmanager-1",
+ "reason": "Scheduled maintenance",
+ "duration": "PT2H"
+ }'
+
+# List all blocked nodes
+curl http://localhost:8081/cluster/blocklist
+
+# Remove a node from blocklist
+curl -X DELETE http://localhost:8081/cluster/blocklist/taskmanager-1
+```
+
+### Using Flink CLI (if available)
+
+```bash
+# Add node to blocklist
+flink cluster blocklist add taskmanager-1 "Maintenance required" --duration 2h
+
+# List blocked nodes
+flink cluster blocklist list
+
+# Remove node from blocklist
+flink cluster blocklist remove taskmanager-1
+```
+
+## Behavior and Limitations
+
+### Blocking Behavior
+- **New allocations**: Blocked nodes will not receive new slot allocations
+- **Existing tasks**: Tasks already running on blocked nodes continue to run normally
+- **Automatic expiration**: Nodes are automatically unblocked after the specified duration
+- **Immediate effect**: Blocking takes effect immediately after the API call
+
+### Limitations
+- **Cluster stability**: Blocking too many nodes may cause resource shortage
+- **Running jobs**: Existing tasks on blocked nodes are not affected
+- **Persistence**: Blocklist state is not persisted across JobManager restarts
+- **Network partitions**: Blocked nodes that become unreachable are handled by Flink's failure detection
+
+### Best Practices
+1. **Gradual blocking**: Block nodes gradually to avoid resource shortage
+2. **Monitor resources**: Ensure sufficient resources remain available after blocking
+3. **Clear reasons**: Provide clear, descriptive reasons for blocking nodes
+4. **Reasonable durations**: Use appropriate block durations based on maintenance needs
Review Comment:
this implies a maintenance use case - is that the main use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873358604
##########
docs/content/docs/ops/management_blocklist.md:
##########
@@ -0,0 +1,218 @@
+---
+title: "Management Blocklist"
+weight: 9
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Management Blocklist
+
+The Management Blocklist is a feature that allows cluster administrators to temporarily block problematic nodes from receiving new task allocations. This is useful for maintenance, troubleshooting, or handling nodes with performance issues without completely removing them from the cluster.
+
+## Overview
+
+The Management Blocklist provides a way to:
+- **Block nodes**: Prevent new slots from being allocated on specific nodes
+- **Unblock nodes**: Allow previously blocked nodes to receive new allocations again
+- **View blocked nodes**: List all currently blocked nodes with their block reasons and expiration times
+- **Automatic cleanup**: Blocked nodes are automatically unblocked after a specified duration
+
+Unlike the [speculative execution blocklist]({{< ref "docs/deployment/speculative_execution" >}}), the Management Blocklist is manually controlled by administrators and is intended for operational management rather than automatic performance optimization.
+
+## Configuration
+
+The Management Blocklist can be configured using the following options:
+
+| Configuration Key | Default | Description |
+|-------------------|---------|-------------|
+| `management.blocklist.enabled` | `false` | Whether to enable the management blocklist feature |
+| `management.blocklist.default-duration` | `1h` | Default duration for blocking nodes when no explicit duration is provided |
+| `management.blocklist.max-duration` | `24h` | Maximum allowed duration for blocking nodes |
+
+Example configuration:
+```yaml
+management:
+ blocklist:
+ enabled: true
+ default-duration: 30min
+ max-duration: 12h
+```
+
+## REST API
+
+The Management Blocklist provides a REST API for programmatic control:
+
+### Add Node to Blocklist
+
+**Endpoint**: `POST /cluster/blocklist`
+
+**Request Body**:
+```json
+{
+ "nodeId": "taskmanager-1",
+ "reason": "Maintenance required",
+ "duration": "PT2H"
+}
+```
+
+**Parameters**:
+- `nodeId` (required): The ID of the node to block
+- `reason` (required): Human-readable reason for blocking the node
+- `duration` (optional): Duration to block the node (ISO 8601 format). If not provided, uses the configured default duration.
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been added to the blocklist"
+}
+```
+
+### Remove Node from Blocklist
+
+**Endpoint**: `DELETE /cluster/blocklist/{nodeId}`
+
+**Parameters**:
+- `nodeId` (path parameter): The ID of the node to unblock
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been removed from the blocklist"
+}
+```
+
+### List Blocked Nodes
+
+**Endpoint**: `GET /cluster/blocklist`
+
+**Response**:
+```json
+{
+ "blockedNodes": [
+ {
+ "nodeId": "taskmanager-1",
+ "cause": "Maintenance required",
+ "blockedAt": "2024-03-15T10:30:00Z",
+ "expiresAt": "2024-03-15T12:30:00Z"
+ },
+ {
+ "nodeId": "taskmanager-3",
+ "cause": "Performance issues",
+ "blockedAt": "2024-03-15T11:00:00Z",
+ "expiresAt": "2024-03-15T13:00:00Z"
+ }
+ ]
+}
+```
+
+## Usage Examples
+
+### Using curl
+
+```bash
+# Add a node to blocklist for 2 hours
+curl -X POST http://localhost:8081/cluster/blocklist \
+ -H "Content-Type: application/json" \
+ -d '{
+ "nodeId": "taskmanager-1",
+ "reason": "Scheduled maintenance",
+ "duration": "PT2H"
+ }'
+
+# List all blocked nodes
+curl http://localhost:8081/cluster/blocklist
+
+# Remove a node from blocklist
+curl -X DELETE http://localhost:8081/cluster/blocklist/taskmanager-1
+```
+
+### Using Flink CLI (if available)
+
+```bash
+# Add node to blocklist
+flink cluster blocklist add taskmanager-1 "Maintenance required" --duration 2h
+
+# List blocked nodes
+flink cluster blocklist list
+
+# Remove node from blocklist
+flink cluster blocklist remove taskmanager-1
+```
+
+## Behavior and Limitations
+
+### Blocking Behavior
+- **New allocations**: Blocked nodes will not receive new slot allocations
+- **Existing tasks**: Tasks already running on blocked nodes continue to run normally
+- **Automatic expiration**: Nodes are automatically unblocked after the specified duration
+- **Immediate effect**: Blocking takes effect immediately after the API call
+
+### Limitations
+- **Cluster stability**: Blocking too many nodes may cause resource shortage
+- **Running jobs**: Existing tasks on blocked nodes are not affected
+- **Persistence**: Blocklist state is not persisted across JobManager restarts
+- **Network partitions**: Blocked nodes that become unreachable are handled by Flink's failure detection
+
+### Best Practices
+1. **Gradual blocking**: Block nodes gradually to avoid resource shortage
+2. **Monitor resources**: Ensure sufficient resources remain available after blocking
Review Comment:
ho do you determine what is sufficient - some guidance would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873352441
##########
docs/content/docs/ops/management_blocklist.md:
##########
@@ -0,0 +1,218 @@
+---
+title: "Management Blocklist"
+weight: 9
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Management Blocklist
+
+The Management Blocklist is a feature that allows cluster administrators to temporarily block problematic nodes from receiving new task allocations. This is useful for maintenance, troubleshooting, or handling nodes with performance issues without completely removing them from the cluster.
+
+## Overview
+
+The Management Blocklist provides a way to:
+- **Block nodes**: Prevent new slots from being allocated on specific nodes
+- **Unblock nodes**: Allow previously blocked nodes to receive new allocations again
+- **View blocked nodes**: List all currently blocked nodes with their block reasons and expiration times
+- **Automatic cleanup**: Blocked nodes are automatically unblocked after a specified duration
+
+Unlike the [speculative execution blocklist]({{< ref "docs/deployment/speculative_execution" >}}), the Management Blocklist is manually controlled by administrators and is intended for operational management rather than automatic performance optimization.
+
+## Configuration
+
+The Management Blocklist can be configured using the following options:
+
+| Configuration Key | Default | Description |
+|-------------------|---------|-------------|
+| `management.blocklist.enabled` | `false` | Whether to enable the management blocklist feature |
+| `management.blocklist.default-duration` | `1h` | Default duration for blocking nodes when no explicit duration is provided |
+| `management.blocklist.max-duration` | `24h` | Maximum allowed duration for blocking nodes |
+
+Example configuration:
+```yaml
+management:
+ blocklist:
+ enabled: true
+ default-duration: 30min
+ max-duration: 12h
+```
+
+## REST API
+
+The Management Blocklist provides a REST API for programmatic control:
+
+### Add Node to Blocklist
+
+**Endpoint**: `POST /cluster/blocklist`
+
+**Request Body**:
+```json
+{
+ "nodeId": "taskmanager-1",
+ "reason": "Maintenance required",
+ "duration": "PT2H"
+}
+```
+
+**Parameters**:
+- `nodeId` (required): The ID of the node to block
+- `reason` (required): Human-readable reason for blocking the node
+- `duration` (optional): Duration to block the node (ISO 8601 format). If not provided, uses the configured default duration.
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been added to the blocklist"
+}
+```
+
+### Remove Node from Blocklist
+
+**Endpoint**: `DELETE /cluster/blocklist/{nodeId}`
+
+**Parameters**:
+- `nodeId` (path parameter): The ID of the node to unblock
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been removed from the blocklist"
+}
+```
+
+### List Blocked Nodes
+
+**Endpoint**: `GET /cluster/blocklist`
+
+**Response**:
+```json
+{
+ "blockedNodes": [
+ {
+ "nodeId": "taskmanager-1",
+ "cause": "Maintenance required",
+ "blockedAt": "2024-03-15T10:30:00Z",
+ "expiresAt": "2024-03-15T12:30:00Z"
+ },
+ {
+ "nodeId": "taskmanager-3",
+ "cause": "Performance issues",
+ "blockedAt": "2024-03-15T11:00:00Z",
+ "expiresAt": "2024-03-15T13:00:00Z"
+ }
+ ]
+}
+```
+
+## Usage Examples
+
+### Using curl
+
+```bash
+# Add a node to blocklist for 2 hours
+curl -X POST http://localhost:8081/cluster/blocklist \
+ -H "Content-Type: application/json" \
+ -d '{
+ "nodeId": "taskmanager-1",
+ "reason": "Scheduled maintenance",
Review Comment:
I am curious why would you block a task manager for scheduled maintenance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873365112
##########
docs/content/docs/ops/management_blocklist.md:
##########
@@ -0,0 +1,218 @@
+---
+title: "Management Blocklist"
+weight: 9
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Management Blocklist
+
+The Management Blocklist is a feature that allows cluster administrators to temporarily block problematic nodes from receiving new task allocations. This is useful for maintenance, troubleshooting, or handling nodes with performance issues without completely removing them from the cluster.
+
+## Overview
+
+The Management Blocklist provides a way to:
+- **Block nodes**: Prevent new slots from being allocated on specific nodes
+- **Unblock nodes**: Allow previously blocked nodes to receive new allocations again
+- **View blocked nodes**: List all currently blocked nodes with their block reasons and expiration times
+- **Automatic cleanup**: Blocked nodes are automatically unblocked after a specified duration
+
+Unlike the [speculative execution blocklist]({{< ref "docs/deployment/speculative_execution" >}}), the Management Blocklist is manually controlled by administrators and is intended for operational management rather than automatic performance optimization.
+
+## Configuration
+
+The Management Blocklist can be configured using the following options:
+
+| Configuration Key | Default | Description |
+|-------------------|---------|-------------|
+| `management.blocklist.enabled` | `false` | Whether to enable the management blocklist feature |
+| `management.blocklist.default-duration` | `1h` | Default duration for blocking nodes when no explicit duration is provided |
+| `management.blocklist.max-duration` | `24h` | Maximum allowed duration for blocking nodes |
+
+Example configuration:
+```yaml
+management:
+ blocklist:
+ enabled: true
+ default-duration: 30min
+ max-duration: 12h
+```
+
+## REST API
+
+The Management Blocklist provides a REST API for programmatic control:
+
+### Add Node to Blocklist
+
+**Endpoint**: `POST /cluster/blocklist`
+
+**Request Body**:
+```json
+{
+ "nodeId": "taskmanager-1",
+ "reason": "Maintenance required",
+ "duration": "PT2H"
+}
+```
+
+**Parameters**:
+- `nodeId` (required): The ID of the node to block
+- `reason` (required): Human-readable reason for blocking the node
+- `duration` (optional): Duration to block the node (ISO 8601 format). If not provided, uses the configured default duration.
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been added to the blocklist"
+}
+```
+
+### Remove Node from Blocklist
+
+**Endpoint**: `DELETE /cluster/blocklist/{nodeId}`
+
+**Parameters**:
+- `nodeId` (path parameter): The ID of the node to unblock
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been removed from the blocklist"
+}
+```
+
+### List Blocked Nodes
+
+**Endpoint**: `GET /cluster/blocklist`
+
+**Response**:
+```json
+{
+ "blockedNodes": [
+ {
+ "nodeId": "taskmanager-1",
+ "cause": "Maintenance required",
+ "blockedAt": "2024-03-15T10:30:00Z",
+ "expiresAt": "2024-03-15T12:30:00Z"
+ },
+ {
+ "nodeId": "taskmanager-3",
+ "cause": "Performance issues",
+ "blockedAt": "2024-03-15T11:00:00Z",
+ "expiresAt": "2024-03-15T13:00:00Z"
+ }
+ ]
+}
+```
+
+## Usage Examples
+
+### Using curl
+
+```bash
+# Add a node to blocklist for 2 hours
+curl -X POST http://localhost:8081/cluster/blocklist \
+ -H "Content-Type: application/json" \
+ -d '{
+ "nodeId": "taskmanager-1",
+ "reason": "Scheduled maintenance",
+ "duration": "PT2H"
+ }'
+
+# List all blocked nodes
+curl http://localhost:8081/cluster/blocklist
+
+# Remove a node from blocklist
+curl -X DELETE http://localhost:8081/cluster/blocklist/taskmanager-1
+```
+
+### Using Flink CLI (if available)
+
+```bash
+# Add node to blocklist
+flink cluster blocklist add taskmanager-1 "Maintenance required" --duration 2h
+
+# List blocked nodes
+flink cluster blocklist list
+
+# Remove node from blocklist
+flink cluster blocklist remove taskmanager-1
+```
+
+## Behavior and Limitations
+
+### Blocking Behavior
+- **New allocations**: Blocked nodes will not receive new slot allocations
+- **Existing tasks**: Tasks already running on blocked nodes continue to run normally
+- **Automatic expiration**: Nodes are automatically unblocked after the specified duration
+- **Immediate effect**: Blocking takes effect immediately after the API call
+
+### Limitations
+- **Cluster stability**: Blocking too many nodes may cause resource shortage
+- **Running jobs**: Existing tasks on blocked nodes are not affected
+- **Persistence**: Blocklist state is not persisted across JobManager restarts
+- **Network partitions**: Blocked nodes that become unreachable are handled by Flink's failure detection
+
+### Best Practices
+1. **Gradual blocking**: Block nodes gradually to avoid resource shortage
+2. **Monitor resources**: Ensure sufficient resources remain available after blocking
+3. **Clear reasons**: Provide clear, descriptive reasons for blocking nodes
+4. **Reasonable durations**: Use appropriate block durations based on maintenance needs
+5. **Coordination**: Coordinate with job owners before blocking nodes with running tasks
Review Comment:
what would a job owner expect in this case, I guess slower through put and potential SLAs not being met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "featzhang (via GitHub)" <gi...@apache.org>.
featzhang closed pull request #27716: [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI
URL: https://github.com/apache/flink/pull/27716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27716:
URL: https://github.com/apache/flink/pull/27716#issuecomment-3979119965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e74fddadc54946bf84d0c3b11784622e243055fb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e74fddadc54946bf84d0c3b11784622e243055fb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e74fddadc54946bf84d0c3b11784622e243055fb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873327706
##########
docs/content/docs/ops/management_blocklist.md:
##########
@@ -0,0 +1,218 @@
+---
+title: "Management Blocklist"
+weight: 9
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Management Blocklist
+
+The Management Blocklist is a feature that allows cluster administrators to temporarily block problematic nodes from receiving new task allocations. This is useful for maintenance, troubleshooting, or handling nodes with performance issues without completely removing them from the cluster.
Review Comment:
it would be useful to give some details on how you might recognise a problematic node and why no receiving new task allocations is useful; I assume there was a scenario which this was reported for. I assume that this would be some sort of process wide issue as we are stopping new slots from being allocated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "featzhang (via GitHub)" <gi...@apache.org>.
featzhang commented on PR #27716:
URL: https://github.com/apache/flink/pull/27716#issuecomment-4064451372
This PR is being closed due to CI failures. Please fix the issues and reopen when ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873302423
##########
flink-runtime/src/test/java/org/apache/flink/runtime/management/blocklist/ManagementBlocklistEdgeCasesTest.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.management.blocklist;
+
+import org.apache.flink.runtime.blocklist.BlockedNode;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.time.Duration;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Tests for edge cases and boundary conditions in management blocklist functionality. */
+public class ManagementBlocklistEdgeCasesTest {
+
+ private DefaultManagementBlocklistHandler handler;
+ private ScheduledExecutorService executorService;
+
+ @BeforeEach
+ public void setUp() {
+ executorService = Executors.newSingleThreadScheduledExecutor();
+ handler =
+ new DefaultManagementBlocklistHandler(
+ ComponentMainThreadExecutorServiceAdapter.forSingleThreadExecutor(
+ executorService),
+ Duration.ofSeconds(1)); // 1 second cleanup interval
+ }
+
+ @Test
+ public void testAddNodeWithNullNodeId() {
+ // Test null nodeId
+ assertThrows(
+ NullPointerException.class,
+ () -> {
+ handler.addBlockedNode(null, "reason", Duration.ofMinutes(5));
+ });
+ }
+
+ @Test
+ public void testAddNodeWithNullReason() {
+ // Test null reason
+ assertThrows(
+ NullPointerException.class,
+ () -> {
+ handler.addBlockedNode("node1", null, Duration.ofMinutes(5));
+ });
+ }
+
+ @Test
+ public void testAddNodeWithNullDuration() {
+ // Test null duration
+ assertThrows(
+ NullPointerException.class,
+ () -> {
+ handler.addBlockedNode("node1", "reason", null);
+ });
+ }
+
+ @Test
+ public void testRemoveNodeWithNullNodeId() {
+ // Test null nodeId for removal
+ assertThrows(
+ NullPointerException.class,
+ () -> {
+ handler.removeBlockedNode(null);
+ });
+ }
+
+ @Test
+ public void testRemoveNonExistentNode() {
+ // Test removing a node that doesn't exist
+ boolean result = handler.removeBlockedNode("non-existent-node");
+ assertFalse(result);
+ }
+
+ @Test
+ public void testIsBlockedWithNullNodeId() {
+ // Test null nodeId for isBlocked check
+ assertThrows(
+ NullPointerException.class,
+ () -> {
+ handler.isNodeBlocked(null);
+ });
+ }
+
+ @Test
+ public void testAddSameNodeMultipleTimes() {
+ // Test adding the same node multiple times
+ String nodeId = "test-node";
+ String reason1 = "first reason";
+ String reason2 = "second reason";
+ Duration duration1 = Duration.ofMinutes(10);
+ Duration duration2 = Duration.ofMinutes(20);
+
+ // Add node first time
+ handler.addBlockedNode(nodeId, reason1, duration1);
+ assertTrue(handler.isNodeBlocked(nodeId));
+
+ // Add same node second time (should update)
+ handler.addBlockedNode(nodeId, reason2, duration2);
+ assertTrue(handler.isNodeBlocked(nodeId));
+
+ // Verify the node exists and has updated information
+ Set<BlockedNode> allNodes = handler.getAllBlockedNodes();
+ assertEquals(1, allNodes.size());
+ BlockedNode node = allNodes.iterator().next();
+ assertEquals(nodeId, node.getNodeId());
+ assertEquals(reason2, node.getCause()); // Should have the updated reason
+ }
+
+ @Test
+ public void testVeryShortDuration() {
+ // Test with very short duration (10 milliseconds)
+ String nodeId = "short-duration-node";
+ Duration shortDuration = Duration.ofMillis(10);
+
+ handler.addBlockedNode(nodeId, "short test", shortDuration);
+ assertTrue(handler.isNodeBlocked(nodeId));
+
+ // Wait a bit and check if it's still blocked (might have expired)
+ try {
+ Thread.sleep(50);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+
+ // The node might have expired by now, but that's expected behavior
+ Set<BlockedNode> allNodes = handler.getAllBlockedNodes();
+ // We don't assert the result here because it depends on timing
+ assertNotNull(allNodes);
+ }
+
+ @Test
+ public void testVeryLongDuration() {
+ // Test with very long duration
+ String nodeId = "long-duration-node";
+ Duration longDuration = Duration.ofDays(365); // 1 year
+
+ handler.addBlockedNode(nodeId, "long test", longDuration);
+ assertTrue(handler.isNodeBlocked(nodeId));
+
+ Set<BlockedNode> allNodes = handler.getAllBlockedNodes();
+ assertEquals(1, allNodes.size());
+ BlockedNode node = allNodes.iterator().next();
+ assertEquals(nodeId, node.getNodeId());
+ assertTrue(
+ node.getEndTimestamp()
+ > System.currentTimeMillis() + Duration.ofDays(300).toMillis());
+ }
+
+ @Test
+ public void testLargeNumberOfNodes() {
+ // Test adding a large number of nodes
+ int nodeCount = 100; // Reduced from 1000 to avoid test timeout
+ Duration duration = Duration.ofMinutes(30);
+
+ for (int i = 0; i < nodeCount; i++) {
+ String nodeId = "node-" + i;
+ handler.addBlockedNode(nodeId, "reason-" + i, duration);
+ }
+
+ // Verify all nodes are blocked
+ for (int i = 0; i < nodeCount; i++) {
+ String nodeId = "node-" + i;
+ assertTrue(handler.isNodeBlocked(nodeId), "Node " + nodeId + " should be blocked");
+ }
+
+ Set<BlockedNode> allNodes = handler.getAllBlockedNodes();
+ assertEquals(nodeCount, allNodes.size());
+ }
+
+ @Test
+ public void testSpecialCharactersInNodeId() {
+ // Test node IDs with special characters
+ String[] specialNodeIds = {
+ "node-with-dashes", "node_with_underscores", "node.with.dots", "node:with:colons"
+ };
+
+ Duration duration = Duration.ofMinutes(10);
+
+ for (String nodeId : specialNodeIds) {
+ handler.addBlockedNode(nodeId, "special char test", duration);
+ assertTrue(
+ handler.isNodeBlocked(nodeId),
+ "Node with special chars should be blocked: " + nodeId);
+ }
+
+ Set<BlockedNode> allNodes = handler.getAllBlockedNodes();
+ assertEquals(specialNodeIds.length, allNodes.size());
+ }
+
+ @Test
+ public void testSpecialCharactersInReason() {
+ // Test reasons with special characters
+ String nodeId = "test-node";
+ String[] specialReasons = {
+ "Reason with spaces and punctuation!",
+ "Reason with unicode: 测试原因",
+ "Reason with newlines:\nSecond line",
+ "Reason with tabs:\tTabbed content",
+ "Reason with quotes: \"quoted\" and 'single quoted'"
+ };
+
+ Duration duration = Duration.ofMinutes(10);
+
+ for (int i = 0; i < specialReasons.length; i++) {
+ String currentNodeId = nodeId + "-" + i;
+ handler.addBlockedNode(currentNodeId, specialReasons[i], duration);
+ assertTrue(handler.isNodeBlocked(currentNodeId));
+ }
+
+ Set<BlockedNode> allNodes = handler.getAllBlockedNodes();
+ assertEquals(specialReasons.length, allNodes.size());
+ }
+
+ @Test
+ public void testBasicAddRemoveOperations() {
+ // Test basic add/remove operations
+ String nodeId = "test-node";
+ Duration duration = Duration.ofMinutes(30);
+
+ // Initially not blocked
+ assertFalse(handler.isNodeBlocked(nodeId));
+
+ // Add node
+ handler.addBlockedNode(nodeId, "test reason", duration);
+ assertTrue(handler.isNodeBlocked(nodeId));
+
+ // Remove node
+ boolean removed = handler.removeBlockedNode(nodeId);
+ assertTrue(removed);
+ assertFalse(handler.isNodeBlocked(nodeId));
+
+ // Try to remove again
+ boolean removedAgain = handler.removeBlockedNode(nodeId);
+ assertFalse(removedAgain);
+ }
+
+ @Test
+ public void testGetAllBlockedNodes() {
+ // Test getAllBlockedNodes method
+ assertTrue(handler.getAllBlockedNodes().isEmpty());
+
+ // Add some nodes
+ handler.addBlockedNode("node1", "reason1", Duration.ofMinutes(10));
+ handler.addBlockedNode("node2", "reason2", Duration.ofMinutes(20));
+ handler.addBlockedNode("node3", "reason3", Duration.ofMinutes(30));
+
+ Set<BlockedNode> allNodes = handler.getAllBlockedNodes();
+ assertEquals(3, allNodes.size());
+
+ // Remove one node
+ handler.removeBlockedNode("node2");
+ allNodes = handler.getAllBlockedNodes();
+ assertEquals(2, allNodes.size());
+ }
+
+ @Test
+ public void testExpiredNodeRemoval() throws InterruptedException {
+ // Test automatic removal of expired nodes
+ String nodeId = "expiring-node";
+ Duration shortDuration = Duration.ofMillis(100);
+
+ handler.addBlockedNode(nodeId, "expiring test", shortDuration);
+ assertTrue(handler.isNodeBlocked(nodeId));
+
+ // Wait for expiration
+ Thread.sleep(200);
+
+ // Manually trigger cleanup (since we can't wait for automatic cleanup)
+ handler.removeExpiredNodes();
Review Comment:
I suggest checking the return nodes are as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873356175
##########
docs/content/docs/ops/management_blocklist.md:
##########
@@ -0,0 +1,218 @@
+---
+title: "Management Blocklist"
+weight: 9
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Management Blocklist
+
+The Management Blocklist is a feature that allows cluster administrators to temporarily block problematic nodes from receiving new task allocations. This is useful for maintenance, troubleshooting, or handling nodes with performance issues without completely removing them from the cluster.
+
+## Overview
+
+The Management Blocklist provides a way to:
+- **Block nodes**: Prevent new slots from being allocated on specific nodes
+- **Unblock nodes**: Allow previously blocked nodes to receive new allocations again
+- **View blocked nodes**: List all currently blocked nodes with their block reasons and expiration times
+- **Automatic cleanup**: Blocked nodes are automatically unblocked after a specified duration
+
+Unlike the [speculative execution blocklist]({{< ref "docs/deployment/speculative_execution" >}}), the Management Blocklist is manually controlled by administrators and is intended for operational management rather than automatic performance optimization.
+
+## Configuration
+
+The Management Blocklist can be configured using the following options:
+
+| Configuration Key | Default | Description |
+|-------------------|---------|-------------|
+| `management.blocklist.enabled` | `false` | Whether to enable the management blocklist feature |
+| `management.blocklist.default-duration` | `1h` | Default duration for blocking nodes when no explicit duration is provided |
+| `management.blocklist.max-duration` | `24h` | Maximum allowed duration for blocking nodes |
+
+Example configuration:
+```yaml
+management:
+ blocklist:
+ enabled: true
+ default-duration: 30min
+ max-duration: 12h
+```
+
+## REST API
+
+The Management Blocklist provides a REST API for programmatic control:
+
+### Add Node to Blocklist
+
+**Endpoint**: `POST /cluster/blocklist`
+
+**Request Body**:
+```json
+{
+ "nodeId": "taskmanager-1",
+ "reason": "Maintenance required",
+ "duration": "PT2H"
+}
+```
+
+**Parameters**:
+- `nodeId` (required): The ID of the node to block
+- `reason` (required): Human-readable reason for blocking the node
+- `duration` (optional): Duration to block the node (ISO 8601 format). If not provided, uses the configured default duration.
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been added to the blocklist"
+}
+```
+
+### Remove Node from Blocklist
+
+**Endpoint**: `DELETE /cluster/blocklist/{nodeId}`
+
+**Parameters**:
+- `nodeId` (path parameter): The ID of the node to unblock
+
+**Response**:
+```json
+{
+ "status": "success",
+ "message": "Node taskmanager-1 has been removed from the blocklist"
+}
+```
+
+### List Blocked Nodes
+
+**Endpoint**: `GET /cluster/blocklist`
+
+**Response**:
+```json
+{
+ "blockedNodes": [
+ {
+ "nodeId": "taskmanager-1",
+ "cause": "Maintenance required",
+ "blockedAt": "2024-03-15T10:30:00Z",
+ "expiresAt": "2024-03-15T12:30:00Z"
+ },
+ {
+ "nodeId": "taskmanager-3",
+ "cause": "Performance issues",
+ "blockedAt": "2024-03-15T11:00:00Z",
+ "expiresAt": "2024-03-15T13:00:00Z"
+ }
+ ]
+}
+```
+
+## Usage Examples
+
+### Using curl
+
+```bash
+# Add a node to blocklist for 2 hours
+curl -X POST http://localhost:8081/cluster/blocklist \
+ -H "Content-Type: application/json" \
+ -d '{
+ "nodeId": "taskmanager-1",
+ "reason": "Scheduled maintenance",
+ "duration": "PT2H"
+ }'
+
+# List all blocked nodes
+curl http://localhost:8081/cluster/blocklist
+
+# Remove a node from blocklist
+curl -X DELETE http://localhost:8081/cluster/blocklist/taskmanager-1
+```
+
+### Using Flink CLI (if available)
+
+```bash
+# Add node to blocklist
+flink cluster blocklist add taskmanager-1 "Maintenance required" --duration 2h
+
+# List blocked nodes
+flink cluster blocklist list
+
+# Remove node from blocklist
+flink cluster blocklist remove taskmanager-1
+```
+
+## Behavior and Limitations
+
+### Blocking Behavior
+- **New allocations**: Blocked nodes will not receive new slot allocations
+- **Existing tasks**: Tasks already running on blocked nodes continue to run normally
+- **Automatic expiration**: Nodes are automatically unblocked after the specified duration
+- **Immediate effect**: Blocking takes effect immediately after the API call
+
+### Limitations
+- **Cluster stability**: Blocking too many nodes may cause resource shortage
+- **Running jobs**: Existing tasks on blocked nodes are not affected
+- **Persistence**: Blocklist state is not persisted across JobManager restarts
+- **Network partitions**: Blocked nodes that become unreachable are handled by Flink's failure detection
+
+### Best Practices
+1. **Gradual blocking**: Block nodes gradually to avoid resource shortage
Review Comment:
how do you know how gradual you need to be - can we give some guidance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873327706
##########
docs/content/docs/ops/management_blocklist.md:
##########
@@ -0,0 +1,218 @@
+---
+title: "Management Blocklist"
+weight: 9
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Management Blocklist
+
+The Management Blocklist is a feature that allows cluster administrators to temporarily block problematic nodes from receiving new task allocations. This is useful for maintenance, troubleshooting, or handling nodes with performance issues without completely removing them from the cluster.
Review Comment:
it would be useful to give some details on how you might recognise a problematic node and why no receiving new task allocations is useful; I assume there was a scenario which this was reported for. I assume that this would be some sort of process wide issue as we are stopping new slots from being allocated.
It would be useful to understand the consequences to the stream oif doing this , I assume there are side effects, like potentially increased back pressure if there are not enough other task managers available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39176][WebUI] Add Node Health status page to Job Manager UI [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #27716:
URL: https://github.com/apache/flink/pull/27716#discussion_r2873313869
##########
flink-runtime/src/main/java/org/apache/flink/runtime/management/blocklist/ManagementBlocklistHandler.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.management.blocklist;
+
+import org.apache.flink.runtime.blocklist.BlockedNode;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Set;
+
+/**
+ * Interface for managing cluster-level node blocklist. This is independent of batch execution
Review Comment:
I suggest detailing here what the intent of the block list capability is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org