Suspicious logic in AllocatedEvaluator.cs

[Andrew Chung <af...@gmail.com>]

Hi,

I've noticed something strange in the method ProcessNewEvaluator() in
AllocatedEvaluator.cs. In the method, we specifically check that the memory
of the request matches the allocated evaluator before removing it from
EvaluatorRequestor.Evaluators. That means that if it never matches, the
Evaluator stays there forever and an the AllocatedEvaluatorHandler is never
called with an AllocatedEvaluator that matches the EvaluatorBatchID. This
is dangerous because resource managers such as YARN, for example, can round
the memory up for the AllocatedEvaluator. It is thus likely that if the
user doesn't know what memory multiple to round it to, the user will never
get an AllocatedEvaluator matching the EvaluatorBatchID, and
EvaluatorRequestor.Evaluators can fill up with pending requests.

Is it possible that I misinterpreted the code? Or is this a real concern?
If this is a real concern, I'll file a JIRA issue.

Thanks,
Andrew

Re: Suspicious logic in AllocatedEvaluator.cs

[Andrew Chung <af...@gmail.com>]

Created REEF-877.

Thanks,
Andrew

On Tue, Oct 13, 2015 at 1:19 PM, Andrew Chung <af...@gmail.com> wrote:

> Hi,
>
> I've noticed something strange in the method ProcessNewEvaluator() in
> AllocatedEvaluator.cs. In the method, we specifically check that the memory
> of the request matches the allocated evaluator before removing it from
> EvaluatorRequestor.Evaluators. That means that if it never matches, the
> Evaluator stays there forever and an the AllocatedEvaluatorHandler is never
> called with an AllocatedEvaluator that matches the EvaluatorBatchID. This
> is dangerous because resource managers such as YARN, for example, can round
> the memory up for the AllocatedEvaluator. It is thus likely that if the
> user doesn't know what memory multiple to round it to, the user will never
> get an AllocatedEvaluator matching the EvaluatorBatchID, and
> EvaluatorRequestor.Evaluators can fill up with pending requests.
>
> Is it possible that I misinterpreted the code? Or is this a real concern?
> If this is a real concern, I'll file a JIRA issue.
>
> Thanks,
> Andrew
>