You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by GitBox <gi...@apache.org> on 2020/11/16 18:20:10 UTC
[GitHub] [samza] lhaiesp opened a new pull request #1441: SAMZA-2601: avoid infinite loop on container allocation
lhaiesp opened a new pull request #1441:
URL: https://github.com/apache/samza/pull/1441
Problem: If YARN does not immediately allocate resources for a container and host affinity is disabled for the job, then the container allocator will spam the logs with messages about the request, even if the request is considered expired. This is because the allocator loops over the pending requests to check if they have been allocated, but there is no delay between loop iterations, so it might just keep checking the same pending request over and over as fast as possible. A metric is also incremented in this flow, so that metric is extremely high, which is probably not the intention.
Changes: Avoid infinite loop by breaking the loop if host affinity is disabled
Tests: new unit test
API changes: N/A
Usage instructions: N/A
Upgrade instructions: None
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] Sanil15 commented on a change in pull request #1441: SAMZA-2601: avoid infinite loop on container allocation
Posted by GitBox <gi...@apache.org>.
Sanil15 commented on a change in pull request #1441:
URL: https://github.com/apache/samza/pull/1441#discussion_r528032821
##########
File path: samza-core/src/main/java/org/apache/samza/clustermanager/ContainerAllocator.java
##########
@@ -234,6 +234,12 @@ void assignResourceRequests() {
if (isRequestExpired(request)) {
updateExpiryMetrics(request);
containerManager.handleExpiredRequest(processorId, preferredHost, request, this, resourceRequestState);
+ // SAMZA-2601: to prevent infinite looping and logs filling up the disk, when host affinity is disabled,
+ // we explicitly break the loop here and the whole process gets retried in run() after allocatorSleepIntervalMs
+ if (!hostAffinityEnabled) {
+ LOG.info("Host affinity is disabled on expired request.");
Review comment:
Replace it with something like this:
LOG.info("Waiting for resources to get allocated for request {}, no retries will be issued since host affinity is disabled", request);
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] lhaiesp merged pull request #1441: SAMZA-2601: avoid infinite loop on container allocation
Posted by GitBox <gi...@apache.org>.
lhaiesp merged pull request #1441:
URL: https://github.com/apache/samza/pull/1441
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org