You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by GitBox <gi...@apache.org> on 2020/04/15 03:31:36 UTC
[GitHub] [samza] rmatharu opened a new pull request #1347: SAMZA-2511 :
Adding logic to handle container stop fail
rmatharu opened a new pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347
Problem: The standby container manager does not handle container-stop-failures.
These events can happen as a result of certificate/authentication issue during the execution of the container-stop. The problem is the standby-container-failover flow relies on a stop-container succeeding and in this case does not complete the failover. This means the active container, for which a failover was initiated, is never started again.
In case of a container-placement action, that runs into container-stop-fail, the action is declared as failed.
Cause: Above.
Fix: The fix is for standby-container-manager to intercept and handle these events by continuing the failover by either selecting another standby container (if one is present i.e., rf > 2) or using a standby host or using any-host.
API changes: None
Upgrade Instructions: None
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] rmatharu merged pull request #1347: SAMZA-2511 : Adding
logic to handle container stop fail
Posted by GitBox <gi...@apache.org>.
rmatharu merged pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] Sanil15 commented on a change in pull request #1347:
SAMZA-2511 : Adding logic to handle container stop fail
Posted by GitBox <gi...@apache.org>.
Sanil15 commented on a change in pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347#discussion_r409215717
##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/StandbyContainerManager.java
##########
@@ -126,6 +126,30 @@ public void handleContainerLaunchFail(String containerID, String resourceID,
}
}
+ /**
+ * Handle the failed stop for a container, based on
+ * Case 1. If it is standby container, continue the failover
+ * Case 2. If it is an active container, then log a warning and return.
+ * @param containerID the ID of the container that has failed
Review comment:
nit: @param containerID logical id ex 0,1,2 of the failed samza container
@param resourceID physical id of the failed container
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] rmatharu commented on a change in pull request #1347:
SAMZA-2511 : Adding logic to handle container stop fail
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347#discussion_r409030737
##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/ContainerManager.java
##########
@@ -233,6 +233,30 @@ void handleContainerLaunchFail(String processorId, String containerId, String pr
}
}
+ /**
+ * Handle the container stop failure for active containers and standby (if enabled).
+ * @param processorId logical id of the container eg 1,2,3
+ * @param containerId last known id of the container deployed
+ * @param containerHost host on which container is requested to be deployed
+ * @param containerAllocator allocator for requesting resources
+ */
+ void handleContainerStopFail(String processorId, String containerId, String containerHost,
+ ContainerAllocator containerAllocator) {
+ if (processorId != null && hasActiveContainerPlacementAction(processorId)) {
+ // Assuming resource acquired on destination host will be relinquished by the containerAllocator,
+ // we mark the placement action as failed, and return.
Review comment:
Isnt the container-stop on the active, issued after the resource has already been acquired?
In that case, if the resource has already been acquired, the resource-request for it has already been met, so it should have already been cancelled?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] Sanil15 commented on a change in pull request #1347:
SAMZA-2511 : Adding logic to handle container stop fail
Posted by GitBox <gi...@apache.org>.
Sanil15 commented on a change in pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347#discussion_r409215078
##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/StandbyContainerManager.java
##########
@@ -126,6 +126,30 @@ public void handleContainerLaunchFail(String containerID, String resourceID,
}
}
+ /**
+ * Handle the failed stop for a container, based on
+ * Case 1. If it is standby container, continue the failover
+ * Case 2. If it is an active container, then log a warning and return.
Review comment:
Please correct the comment here, Case 2. If this is an active container, throw an exception restart the AM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] Sanil15 commented on a change in pull request #1347:
SAMZA-2511 : Adding logic to handle container stop fail
Posted by GitBox <gi...@apache.org>.
Sanil15 commented on a change in pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347#discussion_r408579584
##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/ContainerManager.java
##########
@@ -233,6 +233,30 @@ void handleContainerLaunchFail(String processorId, String containerId, String pr
}
}
+ /**
+ * Handle the container stop failure for active containers and standby (if enabled).
+ * @param processorId logical id of the container eg 1,2,3
+ * @param containerId last known id of the container deployed
+ * @param containerHost host on which container is requested to be deployed
+ * @param containerAllocator allocator for requesting resources
+ */
+ void handleContainerStopFail(String processorId, String containerId, String containerHost,
+ ContainerAllocator containerAllocator) {
+ if (processorId != null && hasActiveContainerPlacementAction(processorId)) {
+ // Assuming resource acquired on destination host will be relinquished by the containerAllocator,
+ // we mark the placement action as failed, and return.
Review comment:
In the case of ContainerPlacement actions, request in the allocator is waiting for active container to stop and since the active container stop fails here, you also want to cancel the resource request tied to the placement action
resourceRequestState.cancelResourceRequest(request);
I believe ContainerPlacementMetadata maintains a set the resource-request that we can read, but only need to cancel the latest one from that set (since that is the one that is active)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] Sanil15 commented on a change in pull request #1347:
SAMZA-2511 : Adding logic to handle container stop fail
Posted by GitBox <gi...@apache.org>.
Sanil15 commented on a change in pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347#discussion_r408576566
##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/ContainerManager.java
##########
@@ -233,6 +233,30 @@ void handleContainerLaunchFail(String processorId, String containerId, String pr
}
}
+ /**
+ * Handle the container stop failure for active containers and standby (if enabled).
+ * @param processorId logical id of the container eg 1,2,3
+ * @param containerId last known id of the container deployed
+ * @param containerHost host on which container is requested to be deployed
+ * @param containerAllocator allocator for requesting resources
+ */
+ void handleContainerStopFail(String processorId, String containerId, String containerHost,
+ ContainerAllocator containerAllocator) {
+ if (processorId != null && hasActiveContainerPlacementAction(processorId)) {
+ // Assuming resource acquired on destination host will be relinquished by the containerAllocator,
Review comment:
ContainerAllocator only relinquishes resources when there are no requests in the queue. It will be worth manually releasing the resource in this case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] rmatharu commented on a change in pull request #1347:
SAMZA-2511 : Adding logic to handle container stop fail
Posted by GitBox <gi...@apache.org>.
rmatharu commented on a change in pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347#discussion_r409032464
##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/ContainerManager.java
##########
@@ -233,6 +233,30 @@ void handleContainerLaunchFail(String processorId, String containerId, String pr
}
}
+ /**
+ * Handle the container stop failure for active containers and standby (if enabled).
+ * @param processorId logical id of the container eg 1,2,3
+ * @param containerId last known id of the container deployed
+ * @param containerHost host on which container is requested to be deployed
+ * @param containerAllocator allocator for requesting resources
+ */
+ void handleContainerStopFail(String processorId, String containerId, String containerHost,
+ ContainerAllocator containerAllocator) {
+ if (processorId != null && hasActiveContainerPlacementAction(processorId)) {
+ // Assuming resource acquired on destination host will be relinquished by the containerAllocator,
Review comment:
There is nothing in the ContainerPlacementMetadata to determine the container-id (not processor-id) of the resources already acquired.
The containerid in this callback is of the container whose stop failed.
Given that there are likely many placement actions in the queue (not just one), the resource acquired may be useful for handling subsequent placement actions that are in the queue.
Stop-fail is rare occurrence, so relying on the allocator to relinquish is perhaps not that bad.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] Sanil15 commented on a change in pull request #1347:
SAMZA-2511 : Adding logic to handle container stop fail
Posted by GitBox <gi...@apache.org>.
Sanil15 commented on a change in pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347#discussion_r409216345
##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/ContainerManager.java
##########
@@ -233,6 +239,28 @@ void handleContainerLaunchFail(String processorId, String containerId, String pr
}
}
+ /**
+ * Handle the container stop failure for active containers and standby (if enabled).
+ * @param processorId logical id of the container eg 1,2,3
+ * @param containerId last known id of the container deployed
+ * @param containerHost host on which container is requested to be deployed
+ * @param containerAllocator allocator for requesting resources
+ */
+ void handleContainerStopFail(String processorId, String containerId, String containerHost,
Review comment:
Please add a todo for adding an integ test for this, I can pick that up later!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] Sanil15 commented on a change in pull request #1347:
SAMZA-2511 : Adding logic to handle container stop fail
Posted by GitBox <gi...@apache.org>.
Sanil15 commented on a change in pull request #1347: SAMZA-2511 : Adding logic to handle container stop fail
URL: https://github.com/apache/samza/pull/1347#discussion_r409212968
##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/ContainerManager.java
##########
@@ -233,6 +239,28 @@ void handleContainerLaunchFail(String processorId, String containerId, String pr
}
}
+ /**
+ * Handle the container stop failure for active containers and standby (if enabled).
+ * @param processorId logical id of the container eg 1,2,3
+ * @param containerId last known id of the container deployed
+ * @param containerHost host on which container is requested to be deployed
+ * @param containerAllocator allocator for requesting resources
+ */
+ void handleContainerStopFail(String processorId, String containerId, String containerHost,
+ ContainerAllocator containerAllocator) {
+ if (processorId != null && hasActiveContainerPlacementAction(processorId)) {
+ // Assuming resource acquired on destination host will be relinquished by the containerAllocator,
+ // We mark the placement action as failed, and return.
+ ContainerPlacementMetadata metaData = getPlacementActionMetadata(processorId).get();
Review comment:
AMRMAsync thread will do a container status update here, allocator thread marks the actions as failed. Please remove the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services