You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@myriad.apache.org by Paul Read <pd...@gmail.com> on 2015/06/09 17:55:42 UTC

StateStore and FineGrainedScaling (FGS)

I'm working on the StateStore which at first blush does not seem that
difficult. However with the approaching FGS I suspect I need to store
additional information.

I assume HA would need to reconstruct the FGS state.

Assuming this is correct the following would need to happen:

1. Re-init the YarnNodeCapacityManager with new OfferLifeCycleManager and
NodeStore
    a. rebuild OfferLifecycleManager
    b. store NodeStore
    c. store offerFeedMap
    d. store consumerOfferMap

2. Possibly store the Pseudo Tasks and reconstruct the
PseudoTaskResourceManager

3. Store and rebuild the SchedulerState

I was originally thinking that the Offers under consideration would need to
be stored but those may be encapsulated in the tasking...correct?

Thanks

Re: StateStore and FineGrainedScaling (FGS)

Posted by Paul Read <pd...@gmail.com>.

Great! Thanks for the heads up.

On Tue, Jun 9, 2015 at 12:16 PM, yuliya Feldman <yufeldman@yahoo.com.invalid
> wrote:

> Paul,
> Swapnil is also closely looking into Myriad HA considering FGS, so it
> would be a good idea for two of you to sync up as he also did some
> investigation of what needs to be in StateStore and what does not need to
> be stored.
> He may not reply right away as he is in Hadoop Summit today.
>       From: Paul Read <pd...@gmail.com>
>  To: dev@myriad.incubator.apache.org
>  Sent: Tuesday, June 9, 2015 8:55 AM
>  Subject: StateStore and FineGrainedScaling (FGS)
>
> I'm working on the StateStore which at first blush does not seem that
> difficult. However with the approaching FGS I suspect I need to store
> additional information.
>
> I assume HA would need to reconstruct the FGS state.
>
> Assuming this is correct the following would need to happen:
>
> 1. Re-init the YarnNodeCapacityManager with new OfferLifeCycleManager and
> NodeStore
>     a. rebuild OfferLifecycleManager
>     b. store NodeStore
>     c. store offerFeedMap
>     d. store consumerOfferMap
>
> 2. Possibly store the Pseudo Tasks and reconstruct the
> PseudoTaskResourceManager
>
> 3. Store and rebuild the SchedulerState
>
> I was originally thinking that the Offers under consideration would need to
> be stored but those may be encapsulated in the tasking...correct?
>
> Thanks
>
>
>
>

Re: StateStore and FineGrainedScaling (FGS)

Posted by yuliya Feldman <yu...@yahoo.com.INVALID>.

Paul,
Swapnil is also closely looking into Myriad HA considering FGS, so it would be a good idea for two of you to sync up as he also did some investigation of what needs to be in StateStore and what does not need to be stored.
He may not reply right away as he is in Hadoop Summit today.
      From: Paul Read <pd...@gmail.com>
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, June 9, 2015 8:55 AM
 Subject: StateStore and FineGrainedScaling (FGS)

I'm working on the StateStore which at first blush does not seem that
difficult. However with the approaching FGS I suspect I need to store
additional information.

I assume HA would need to reconstruct the FGS state.

Assuming this is correct the following would need to happen:

1. Re-init the YarnNodeCapacityManager with new OfferLifeCycleManager and
NodeStore
    a. rebuild OfferLifecycleManager
    b. store NodeStore
    c. store offerFeedMap
    d. store consumerOfferMap

2. Possibly store the Pseudo Tasks and reconstruct the
PseudoTaskResourceManager

3. Store and rebuild the SchedulerState

I was originally thinking that the Offers under consideration would need to
be stored but those may be encapsulated in the tasking...correct?

Thanks