You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Ashvin A <aa...@gmail.com> on 2016/03/22 00:18:55 UTC
Map EvaluatorRequest to AllocatedEvaluator
Hi,
I am wondering if there is a way I could assign an identifier while
submitting a EvaluatorRequest and get the identifier back when
AllocatedEvaluator associated to the original request is created.
I am trying to use REEF to deploy a YARN app. The driver is submitting
heterogenous container requests based on core and memory requirements. Each
request corresponds to a specific task. For eg, say two containers are
requested:
1) Task T1 -> Req1 (1 core, 1GB)
2) Task T2 -> Req2 (1 core, 5 GB)
Assume REEF creates evaluator E1 for Req1 and E2 for Req2. Now I would like
to launch task T1 on E1 and T2 on E2. So I need a way to map Evaluator to
requests. I am not sure what is the best way to to do this. While creating
the request, I can only set memory and number of cores. There is no setter
for ID. Any suggestions?
Thanks,
Ashvin
Re: Map EvaluatorRequest to AllocatedEvaluator
Posted by Markus Weimer <ma...@weimo.de>.
On Mon, Mar 21, 2016 at 4:43 PM, Ashvin A <aa...@gmail.com> wrote:
> Is there a REEF jira I could watch?
I don't think so. Maybe we should check whether YARN has one?
> And are there any known workarounds?
As you mentioned, you can submit sequentially. When you have vastly
different requests, you can send the requests all at once, and then
disambiguate in the `AllocatedEvaluator` handler. This works when you
have one kind of container with let's say 4 cores and another one with
1 core. However, YARN is allowed to "round up" to the nearest
container size. If that is 4 cores on a cluster you use, it will
always give you 4 core containers, even if you request 1 core...
In REEF.NET, there is a (ill-fated, IMHO) attempt at codifying such a
matchmaking approach. But for the reasons outlined here, it is also
doomed to fail in much the same ways :(
Markus
Re: Map EvaluatorRequest to AllocatedEvaluator
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
this is one of the most common feature requests. Sadly, we cannot
really solve it in the REEF layer: YARN doesn't allow us to associate
a tag with a request. Once that is possible, it is easy to add the
support in REEF.
Markus
On Mon, Mar 21, 2016 at 4:18 PM, Ashvin A <aa...@gmail.com> wrote:
> Hi,
>
> I am wondering if there is a way I could assign an identifier while
> submitting a EvaluatorRequest and get the identifier back when
> AllocatedEvaluator associated to the original request is created.
>
> I am trying to use REEF to deploy a YARN app. The driver is submitting
> heterogenous container requests based on core and memory requirements. Each
> request corresponds to a specific task. For eg, say two containers are
> requested:
> 1) Task T1 -> Req1 (1 core, 1GB)
> 2) Task T2 -> Req2 (1 core, 5 GB)
>
> Assume REEF creates evaluator E1 for Req1 and E2 for Req2. Now I would like
> to launch task T1 on E1 and T2 on E2. So I need a way to map Evaluator to
> requests. I am not sure what is the best way to to do this. While creating
> the request, I can only set memory and number of cores. There is no setter
> for ID. Any suggestions?
>
> Thanks,
> Ashvin