You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Reed Umbrasas <ry...@microsoft.com.INVALID> on 2018/01/30 22:34:31 UTC
What's the purpose of NodeDescriptionEvent and ResourceCatalog
Hi,
I'm digging into REEF's driver side code and found that every time a node is allocated, a NodeDescriptorEvent is fired which adds a record to the ResourceCatalog (implemented by the ResourceCatalogImpl). However, I couldn't see where that information is used. Does anyone know if the ResourceCatalog is used?
I.e.:
1. Yarn Container manager fires NodeDescriptorEvent: https://github.com/apache/reef/blob/master/lang/java/reef-runtime-yarn/src/main/java/org/apache/reef/runtime/yarn/driver/REEFEventHandlers.java#L64
2. Get's handled by the NodeDescriptorHandler: https://github.com/apache/reef/blob/master/lang/java/reef-common/src/main/java/org/apache/reef/runtime/common/driver/resourcemanager/NodeDescriptorHandler.java#L43
3. Gets handled by the ResourceCatalogImpl: https://github.com/apache/reef/blob/master/lang/java/reef-common/src/main/java/org/apache/reef/runtime/common/driver/catalog/ResourceCatalogImpl.java#L74
4. I don't see how (and if) the NodeDescriptorEvents are accessed.
Thanks,
Reed
Re: What's the purpose of NodeDescriptionEvent and ResourceCatalog
Posted by Markus Weimer <ma...@weimo.de>.
Those APIs were meant to create a uniform API to understand the cluster
structure. As it turns out, neither YARN nor any other resource manager
actually provides the needed information to populate them. Hence, I'd
ignore them for the work on the Azure Batch Runtime.
Markus
On Tue, Jan 30, 2018 at 2:34 PM, Reed Umbrasas <
ryumbra@microsoft.com.invalid> wrote:
> Hi,
>
> I'm digging into REEF's driver side code and found that every time a node
> is allocated, a NodeDescriptorEvent is fired which adds a record to the
> ResourceCatalog (implemented by the ResourceCatalogImpl). However, I
> couldn't see where that information is used. Does anyone know if the
> ResourceCatalog is used?
>
> I.e.:
>
>
> 1. Yarn Container manager fires NodeDescriptorEvent:
> https://github.com/apache/reef/blob/master/lang/java/
> reef-runtime-yarn/src/main/java/org/apache/reef/runtime/
> yarn/driver/REEFEventHandlers.java#L64
> 2. Get's handled by the NodeDescriptorHandler:
> https://github.com/apache/reef/blob/master/lang/java/
> reef-common/src/main/java/org/apache/reef/runtime/common/
> driver/resourcemanager/NodeDescriptorHandler.java#L43
> 3. Gets handled by the ResourceCatalogImpl: https://github.com/apache/
> reef/blob/master/lang/java/reef-common/src/main/java/org/
> apache/reef/runtime/common/driver/catalog/ResourceCatalogImpl.java#L74
> 4. I don't see how (and if) the NodeDescriptorEvents are accessed.
>
> Thanks,
> Reed
>
Re: What's the purpose of NodeDescriptionEvent and ResourceCatalog
Posted by Sergiy Matusevych <se...@gmail.com>.
Hi Reed,
Hmm, let me take a look. I'll let you know soon.
Stay tuned!
Sergiy.
On Tue, Jan 30, 2018 at 2:34 PM, Reed Umbrasas <
ryumbra@microsoft.com.invalid> wrote:
> Hi,
>
> I'm digging into REEF's driver side code and found that every time a node
> is allocated, a NodeDescriptorEvent is fired which adds a record to the
> ResourceCatalog (implemented by the ResourceCatalogImpl). However, I
> couldn't see where that information is used. Does anyone know if the
> ResourceCatalog is used?
>
> I.e.:
>
>
> 1. Yarn Container manager fires NodeDescriptorEvent:
> https://github.com/apache/reef/blob/master/lang/java/
> reef-runtime-yarn/src/main/java/org/apache/reef/runtime/
> yarn/driver/REEFEventHandlers.java#L64
> 2. Get's handled by the NodeDescriptorHandler:
> https://github.com/apache/reef/blob/master/lang/java/
> reef-common/src/main/java/org/apache/reef/runtime/common/
> driver/resourcemanager/NodeDescriptorHandler.java#L43
> 3. Gets handled by the ResourceCatalogImpl: https://github.com/apache/
> reef/blob/master/lang/java/reef-common/src/main/java/org/
> apache/reef/runtime/common/driver/catalog/ResourceCatalogImpl.java#L74
> 4. I don't see how (and if) the NodeDescriptorEvents are accessed.
>
> Thanks,
> Reed
>