You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Andrew Chung <af...@gmail.com> on 2015/07/27 20:32:40 UTC
Breaking backward compatibility in driver restart configurations
Hi All,
I will be changing DriverConfiguration by separating it into two
configuration modules:
1. The runtime-dependent configuration module DriverRestartConfiguration
2. The original configuration module DriverConfiguration, which will not
have any restart parameters.
In addition, there will be a change in the parameter DriverRestartHandler,
which will change from Eventhandler<StartTime> to
Set<EventHandler<StartTime>>, and there will be a another new
ConfigurationModule, DriverRestartHandlersConfiguration, for driver
restarts to allow the users to bind the optional handlers multiple times.
Note that the RestartHandler in DriverRestartConfiguration is required but
the RestartHandler in DriverRestartHandlersConfiguration will be optional.
Is this okay for everyone? I presume no one is using this feature yet, as I
have tested it on an HDInsight cluster and confirmed that it doesn't work.
Thanks,
Andrew
Re: Breaking backward compatibility in driver restart configurations
Posted by Yingda Chen <yd...@gmail.com>.
the proposal looks good to me.
-yingda
On Tue, Jul 28, 2015 at 5:29 PM, Andrew Chung <af...@gmail.com> wrote:
> Are there any objections to this plan? Would like to continue development
> tomorrow.
>
> Thanks,
> Andrew
>
> On Mon, Jul 27, 2015 at 10:50 PM, Yingda Chen <yd...@gmail.com> wrote:
>
> > sure. delegating restart-related logic to another configuration module
> > seems like a fine idea to me.
> >
> > RM restart was not within the scope of original driver-HA work. i don't
> > believe it was supported by Yarn back then, good to know it is supported
> > now and we are taking it to the next level :)
> >
> > On Mon, Jul 27, 2015 at 10:38 PM, Andrew Chung <af...@gmail.com>
> > wrote:
> >
> > > Hi Yingda,
> > >
> > > I think you may be right in that it works in the general case where
> only
> > > the AM dies; however, when I tested the RM restart case with a modified
> > > HelloREEF task with preserveContainersAcrossAppAttempts (modified the
> > tasks
> > > to be long-running), something didn't work correctly, and the driver
> > > detected that all evaluators have turned idle and finished the
> > application.
> > > I have attached the logs on the restarted driver to this email. I can
> > run a
> > > few more tests with different variations, but there are probably some
> > edge
> > > cases that weren't covered before.
> > >
> > > I do think it's still a good idea to delegate restart handlers and the
> > > evaluator preservation logic to another configuration module though.
> What
> > > do you think?
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On Mon, Jul 27, 2015 at 10:03 PM, Yingda Chen <yd...@gmail.com>
> wrote:
> > >
> > >> the (old) driver HA works roughly with the following logic:
> > >>
> > >> 1. when driver HA is enabled, driver will record the come-and-go of
> > every
> > >> container on some hdfs file, so that at any time, there is record of
> how
> > >> many containers are currently alive. in addition, driver will
> maintain a
> > >> http server on which it keeps its connectivity information (ip and
> port
> > >> etc.), the url is formed in a fixed format with application id.
> > >> 2. assume that we have yarn support for container-retention switched
> > >> on(default is off), when AM (driver) fails, yarn would not kill the
> > >> containers associated with the AM, instead the containers will be kept
> > >> alive.
> > >> 3. the AM will be restarted after failure by RM (there should be a
> > config
> > >> for it too), and once it is restarted, it will look check with RM to
> see
> > >> if
> > >> there is any previous containers associated with it, if yes, it will
> > know
> > >> that it is reborn from fire and enter restart mode. at the same time,
> it
> > >> will also start the http-server and put its connectivity information
> > >> there.
> > >> 4. for the Evaluators, when connection to driver is lost (after
> certain
> > >> number of retires), evaluator will assume the driver has failed, and
> > will
> > >> then begin to ping the known driver http address (properly formatted
> > with
> > >> application id) periodically. once the ping goes through, it knows
> that
> > >> the
> > >> driver has restarted and can then get the ip and port information of
> the
> > >> restarted driver. evaluator will then try to reconnect/register with
> the
> > >> restarted driver
> > >> 5. back to the (restarted) driver, it will be expecting evaluators to
> > >> reconnect, and will do sanity check on every evaluator that tries to
> > >> reconnect via container id (note that driver can get the list of
> > expected
> > >> evaluator ids from its previous life through RM). once an expected
> > >> evaluator's reconnect request arrive, driver will do handshake and
> > >> re-establish connection.
> > >> 6. once all evaluators have successfully checked in, driver will deem
> > the
> > >> restart completed successfully.
> > >>
> > >> typing this off the top of my head... it should cover the outline of
> the
> > >> driver restart logic, there are other details and/or corner cases, for
> > >> example if evaluator(s) failed during driver's down time, the restart
> > >> logic
> > >> will be able to tell (some of the evaluators did not check back in),
> and
> > >> driver will fire FailedEvaluatorHandler on those evaluators. All cases
> > >> like
> > >> that are probably better illustrated by reading codes, i agree we
> could
> > >> have done better documentation though :)
> > >>
> > >>
> > >> markus was involved in the design and discussion during
> implementation,
> > he
> > >> should be able to chip in as well.
> > >>
> > >> -yingda
> > >>
> > >>
> > >> On Mon, Jul 27, 2015 at 5:40 PM, Andrew Chung <af...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Byung-Gon,
> > >> >
> > >> > As of now, there are still no architectural changes on driver
> > >> > restartability other than configuration changes.
> > >> >
> > >> > The process will be broken down into (at least) 3 steps:
> > >> > 1. Move restart configurations out of DriverConfiguration, since
> they
> > >> are
> > >> > not dependent on each other. Here, we will have a
> > >> > DriverRestartConfiguration as well as a
> > >> DriverRestartHandlersConfiguration.
> > >> > Parameters in DriverRestartConfiguration can only be bound once,
> while
> > >> > parameters in DriverRestartHandlersConfiguration can be bound
> multiple
> > >> > times, since restart handlers are all modified to be sets of
> handlers.
> > >> The
> > >> > RestartHandler in DriverRestartConfiguration is required, but the
> > >> > RestartHandler in DriverRestartHandlersConfiguration is not
> required,
> > >> since
> > >> > we need to guarantee at least one RestartHandler to be bound in
> order
> > >> for
> > >> > restarts to be functional.
> > >> > 2. Implement runtime specific evaluator preserving logic. I will
> only
> > >> > implement the YARN/HDInsight portion since I don't yet have
> experience
> > >> on
> > >> > Mesos. The new proposed way to preserve evaluators and retrieve them
> > on
> > >> > restarted jobs is to have an injectable* runtime-specific*
> > >> > DriverRestartManager that handles detecting restarts and recording
> and
> > >> > retrieving evaluator IDs on restart. Currently, the functionality is
> > >> > completely embedded into YarnContainerManager, but I would like to
> > make
> > >> the
> > >> > design more modular and configurable by delegating it out to another
> > >> > object. DriverRestartManager functions will be called by special
> > >> > ServiceEventHandlers that are bound in a restart configuration
> > >> mentioned in
> > >> > step 1.
> > >> > 3. Implement the actual restart architecture logic. I am not yet
> clear
> > >> on
> > >> > why the current restart mechanism for YARN is not working, but I
> will
> > >> get
> > >> > more details once steps 1 and 2 are done, and will create a
> > >> > proposal/diagram for it.
> > >> >
> > >> > The PR here:
> > >> > https://github.com/apache/incubator-reef/pull/319
> > >> > is for step 1.
> > >> >
> > >> > Please let me know if you would like clarification on anything.
> > >> >
> > >> > Thanks,
> > >> > Andrew
> > >> >
> > >> > On Mon, Jul 27, 2015 at 4:01 PM, Byung-Gon Chun <bg...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > If there is a design doc on driver restartability, could you share
> > >> one?
> > >> > > It'd be helpful to have some diagram. (ref.
> > >> > > https://github.com/apache/incubator-reef/pull/310)
> > >> > >
> > >> > > Thanks.
> > >> > > -Gon
> > >> > >
> > >> > > On Tue, Jul 28, 2015 at 7:31 AM, Byung-Gon Chun <bgchun@gmail.com
> >
> > >> > wrote:
> > >> > >
> > >> > > > Hi Andrew,
> > >> > > >
> > >> > > > Could you explain the rationale of your design a bit more in
> > detail?
> > >> > > >
> > >> > > > Thanks.
> > >> > > > -Gon
> > >> > > >
> > >> > > > On Tue, Jul 28, 2015 at 3:32 AM, Andrew Chung <
> > afchung90@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > >> Hi All,
> > >> > > >>
> > >> > > >> I will be changing DriverConfiguration by separating it into
> two
> > >> > > >> configuration modules:
> > >> > > >> 1. The runtime-dependent configuration module
> > >> > DriverRestartConfiguration
> > >> > > >> 2. The original configuration module DriverConfiguration, which
> > >> will
> > >> > not
> > >> > > >> have any restart parameters.
> > >> > > >>
> > >> > > >> In addition, there will be a change in the parameter
> > >> > > DriverRestartHandler,
> > >> > > >> which will change from Eventhandler<StartTime> to
> > >> > > >> Set<EventHandler<StartTime>>, and there will be a another new
> > >> > > >> ConfigurationModule, DriverRestartHandlersConfiguration, for
> > driver
> > >> > > >> restarts to allow the users to bind the optional handlers
> > multiple
> > >> > > times.
> > >> > > >> Note that the RestartHandler in DriverRestartConfiguration is
> > >> required
> > >> > > but
> > >> > > >> the RestartHandler in DriverRestartHandlersConfiguration will
> be
> > >> > > optional.
> > >> > > >>
> > >> > > >> Is this okay for everyone? I presume no one is using this
> feature
> > >> yet,
> > >> > > as
> > >> > > >> I
> > >> > > >> have tested it on an HDInsight cluster and confirmed that it
> > >> doesn't
> > >> > > work.
> > >> > > >>
> > >> > > >> Thanks,
> > >> > > >> Andrew
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > Byung-Gon Chun
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Byung-Gon Chun
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Breaking backward compatibility in driver restart configurations
Posted by Andrew Chung <af...@gmail.com>.
Great, thanks for the feedback everyone!
On Tue, Jul 28, 2015 at 11:08 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
> Thanks for the detailed explanation, Andrew and Yingda.
> The plan sounds good to me, too.
>
> -Gon
>
>
> On Wed, Jul 29, 2015 at 12:41 PM, Markus Weimer <ma...@weimo.de> wrote:
>
> > On 2015-07-28 17:29, Andrew Chung wrote:
> > > Are there any objections to this plan?
> >
> > None have come up. We'll consider the PR[0] to be unblocked.
> >
> > > Would like to continue development tomorrow.
> >
> > Sounds good. I just did a pass over the PR.
> >
> > Markus
> >
> > [0]: https://github.com/apache/incubator-reef/pull/319
> >
>
>
>
> --
> Byung-Gon Chun
>
Re: Breaking backward compatibility in driver restart configurations
Posted by Byung-Gon Chun <bg...@gmail.com>.
Thanks for the detailed explanation, Andrew and Yingda.
The plan sounds good to me, too.
-Gon
On Wed, Jul 29, 2015 at 12:41 PM, Markus Weimer <ma...@weimo.de> wrote:
> On 2015-07-28 17:29, Andrew Chung wrote:
> > Are there any objections to this plan?
>
> None have come up. We'll consider the PR[0] to be unblocked.
>
> > Would like to continue development tomorrow.
>
> Sounds good. I just did a pass over the PR.
>
> Markus
>
> [0]: https://github.com/apache/incubator-reef/pull/319
>
--
Byung-Gon Chun
Re: Breaking backward compatibility in driver restart configurations
Posted by Markus Weimer <ma...@weimo.de>.
On 2015-07-28 17:29, Andrew Chung wrote:
> Are there any objections to this plan?
None have come up. We'll consider the PR[0] to be unblocked.
> Would like to continue development tomorrow.
Sounds good. I just did a pass over the PR.
Markus
[0]: https://github.com/apache/incubator-reef/pull/319
Re: Breaking backward compatibility in driver restart configurations
Posted by Andrew Chung <af...@gmail.com>.
Are there any objections to this plan? Would like to continue development
tomorrow.
Thanks,
Andrew
On Mon, Jul 27, 2015 at 10:50 PM, Yingda Chen <yd...@gmail.com> wrote:
> sure. delegating restart-related logic to another configuration module
> seems like a fine idea to me.
>
> RM restart was not within the scope of original driver-HA work. i don't
> believe it was supported by Yarn back then, good to know it is supported
> now and we are taking it to the next level :)
>
> On Mon, Jul 27, 2015 at 10:38 PM, Andrew Chung <af...@gmail.com>
> wrote:
>
> > Hi Yingda,
> >
> > I think you may be right in that it works in the general case where only
> > the AM dies; however, when I tested the RM restart case with a modified
> > HelloREEF task with preserveContainersAcrossAppAttempts (modified the
> tasks
> > to be long-running), something didn't work correctly, and the driver
> > detected that all evaluators have turned idle and finished the
> application.
> > I have attached the logs on the restarted driver to this email. I can
> run a
> > few more tests with different variations, but there are probably some
> edge
> > cases that weren't covered before.
> >
> > I do think it's still a good idea to delegate restart handlers and the
> > evaluator preservation logic to another configuration module though. What
> > do you think?
> >
> > Thanks,
> > Andrew
> >
> > On Mon, Jul 27, 2015 at 10:03 PM, Yingda Chen <yd...@gmail.com> wrote:
> >
> >> the (old) driver HA works roughly with the following logic:
> >>
> >> 1. when driver HA is enabled, driver will record the come-and-go of
> every
> >> container on some hdfs file, so that at any time, there is record of how
> >> many containers are currently alive. in addition, driver will maintain a
> >> http server on which it keeps its connectivity information (ip and port
> >> etc.), the url is formed in a fixed format with application id.
> >> 2. assume that we have yarn support for container-retention switched
> >> on(default is off), when AM (driver) fails, yarn would not kill the
> >> containers associated with the AM, instead the containers will be kept
> >> alive.
> >> 3. the AM will be restarted after failure by RM (there should be a
> config
> >> for it too), and once it is restarted, it will look check with RM to see
> >> if
> >> there is any previous containers associated with it, if yes, it will
> know
> >> that it is reborn from fire and enter restart mode. at the same time, it
> >> will also start the http-server and put its connectivity information
> >> there.
> >> 4. for the Evaluators, when connection to driver is lost (after certain
> >> number of retires), evaluator will assume the driver has failed, and
> will
> >> then begin to ping the known driver http address (properly formatted
> with
> >> application id) periodically. once the ping goes through, it knows that
> >> the
> >> driver has restarted and can then get the ip and port information of the
> >> restarted driver. evaluator will then try to reconnect/register with the
> >> restarted driver
> >> 5. back to the (restarted) driver, it will be expecting evaluators to
> >> reconnect, and will do sanity check on every evaluator that tries to
> >> reconnect via container id (note that driver can get the list of
> expected
> >> evaluator ids from its previous life through RM). once an expected
> >> evaluator's reconnect request arrive, driver will do handshake and
> >> re-establish connection.
> >> 6. once all evaluators have successfully checked in, driver will deem
> the
> >> restart completed successfully.
> >>
> >> typing this off the top of my head... it should cover the outline of the
> >> driver restart logic, there are other details and/or corner cases, for
> >> example if evaluator(s) failed during driver's down time, the restart
> >> logic
> >> will be able to tell (some of the evaluators did not check back in), and
> >> driver will fire FailedEvaluatorHandler on those evaluators. All cases
> >> like
> >> that are probably better illustrated by reading codes, i agree we could
> >> have done better documentation though :)
> >>
> >>
> >> markus was involved in the design and discussion during implementation,
> he
> >> should be able to chip in as well.
> >>
> >> -yingda
> >>
> >>
> >> On Mon, Jul 27, 2015 at 5:40 PM, Andrew Chung <af...@gmail.com>
> >> wrote:
> >>
> >> > Hi Byung-Gon,
> >> >
> >> > As of now, there are still no architectural changes on driver
> >> > restartability other than configuration changes.
> >> >
> >> > The process will be broken down into (at least) 3 steps:
> >> > 1. Move restart configurations out of DriverConfiguration, since they
> >> are
> >> > not dependent on each other. Here, we will have a
> >> > DriverRestartConfiguration as well as a
> >> DriverRestartHandlersConfiguration.
> >> > Parameters in DriverRestartConfiguration can only be bound once, while
> >> > parameters in DriverRestartHandlersConfiguration can be bound multiple
> >> > times, since restart handlers are all modified to be sets of handlers.
> >> The
> >> > RestartHandler in DriverRestartConfiguration is required, but the
> >> > RestartHandler in DriverRestartHandlersConfiguration is not required,
> >> since
> >> > we need to guarantee at least one RestartHandler to be bound in order
> >> for
> >> > restarts to be functional.
> >> > 2. Implement runtime specific evaluator preserving logic. I will only
> >> > implement the YARN/HDInsight portion since I don't yet have experience
> >> on
> >> > Mesos. The new proposed way to preserve evaluators and retrieve them
> on
> >> > restarted jobs is to have an injectable* runtime-specific*
> >> > DriverRestartManager that handles detecting restarts and recording and
> >> > retrieving evaluator IDs on restart. Currently, the functionality is
> >> > completely embedded into YarnContainerManager, but I would like to
> make
> >> the
> >> > design more modular and configurable by delegating it out to another
> >> > object. DriverRestartManager functions will be called by special
> >> > ServiceEventHandlers that are bound in a restart configuration
> >> mentioned in
> >> > step 1.
> >> > 3. Implement the actual restart architecture logic. I am not yet clear
> >> on
> >> > why the current restart mechanism for YARN is not working, but I will
> >> get
> >> > more details once steps 1 and 2 are done, and will create a
> >> > proposal/diagram for it.
> >> >
> >> > The PR here:
> >> > https://github.com/apache/incubator-reef/pull/319
> >> > is for step 1.
> >> >
> >> > Please let me know if you would like clarification on anything.
> >> >
> >> > Thanks,
> >> > Andrew
> >> >
> >> > On Mon, Jul 27, 2015 at 4:01 PM, Byung-Gon Chun <bg...@gmail.com>
> >> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > If there is a design doc on driver restartability, could you share
> >> one?
> >> > > It'd be helpful to have some diagram. (ref.
> >> > > https://github.com/apache/incubator-reef/pull/310)
> >> > >
> >> > > Thanks.
> >> > > -Gon
> >> > >
> >> > > On Tue, Jul 28, 2015 at 7:31 AM, Byung-Gon Chun <bg...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Hi Andrew,
> >> > > >
> >> > > > Could you explain the rationale of your design a bit more in
> detail?
> >> > > >
> >> > > > Thanks.
> >> > > > -Gon
> >> > > >
> >> > > > On Tue, Jul 28, 2015 at 3:32 AM, Andrew Chung <
> afchung90@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > >> Hi All,
> >> > > >>
> >> > > >> I will be changing DriverConfiguration by separating it into two
> >> > > >> configuration modules:
> >> > > >> 1. The runtime-dependent configuration module
> >> > DriverRestartConfiguration
> >> > > >> 2. The original configuration module DriverConfiguration, which
> >> will
> >> > not
> >> > > >> have any restart parameters.
> >> > > >>
> >> > > >> In addition, there will be a change in the parameter
> >> > > DriverRestartHandler,
> >> > > >> which will change from Eventhandler<StartTime> to
> >> > > >> Set<EventHandler<StartTime>>, and there will be a another new
> >> > > >> ConfigurationModule, DriverRestartHandlersConfiguration, for
> driver
> >> > > >> restarts to allow the users to bind the optional handlers
> multiple
> >> > > times.
> >> > > >> Note that the RestartHandler in DriverRestartConfiguration is
> >> required
> >> > > but
> >> > > >> the RestartHandler in DriverRestartHandlersConfiguration will be
> >> > > optional.
> >> > > >>
> >> > > >> Is this okay for everyone? I presume no one is using this feature
> >> yet,
> >> > > as
> >> > > >> I
> >> > > >> have tested it on an HDInsight cluster and confirmed that it
> >> doesn't
> >> > > work.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> Andrew
> >> > > >>
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Byung-Gon Chun
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Byung-Gon Chun
> >> > >
> >> >
> >>
> >
> >
>
Re: Breaking backward compatibility in driver restart configurations
Posted by Yingda Chen <yd...@gmail.com>.
sure. delegating restart-related logic to another configuration module
seems like a fine idea to me.
RM restart was not within the scope of original driver-HA work. i don't
believe it was supported by Yarn back then, good to know it is supported
now and we are taking it to the next level :)
On Mon, Jul 27, 2015 at 10:38 PM, Andrew Chung <af...@gmail.com> wrote:
> Hi Yingda,
>
> I think you may be right in that it works in the general case where only
> the AM dies; however, when I tested the RM restart case with a modified
> HelloREEF task with preserveContainersAcrossAppAttempts (modified the tasks
> to be long-running), something didn't work correctly, and the driver
> detected that all evaluators have turned idle and finished the application.
> I have attached the logs on the restarted driver to this email. I can run a
> few more tests with different variations, but there are probably some edge
> cases that weren't covered before.
>
> I do think it's still a good idea to delegate restart handlers and the
> evaluator preservation logic to another configuration module though. What
> do you think?
>
> Thanks,
> Andrew
>
> On Mon, Jul 27, 2015 at 10:03 PM, Yingda Chen <yd...@gmail.com> wrote:
>
>> the (old) driver HA works roughly with the following logic:
>>
>> 1. when driver HA is enabled, driver will record the come-and-go of every
>> container on some hdfs file, so that at any time, there is record of how
>> many containers are currently alive. in addition, driver will maintain a
>> http server on which it keeps its connectivity information (ip and port
>> etc.), the url is formed in a fixed format with application id.
>> 2. assume that we have yarn support for container-retention switched
>> on(default is off), when AM (driver) fails, yarn would not kill the
>> containers associated with the AM, instead the containers will be kept
>> alive.
>> 3. the AM will be restarted after failure by RM (there should be a config
>> for it too), and once it is restarted, it will look check with RM to see
>> if
>> there is any previous containers associated with it, if yes, it will know
>> that it is reborn from fire and enter restart mode. at the same time, it
>> will also start the http-server and put its connectivity information
>> there.
>> 4. for the Evaluators, when connection to driver is lost (after certain
>> number of retires), evaluator will assume the driver has failed, and will
>> then begin to ping the known driver http address (properly formatted with
>> application id) periodically. once the ping goes through, it knows that
>> the
>> driver has restarted and can then get the ip and port information of the
>> restarted driver. evaluator will then try to reconnect/register with the
>> restarted driver
>> 5. back to the (restarted) driver, it will be expecting evaluators to
>> reconnect, and will do sanity check on every evaluator that tries to
>> reconnect via container id (note that driver can get the list of expected
>> evaluator ids from its previous life through RM). once an expected
>> evaluator's reconnect request arrive, driver will do handshake and
>> re-establish connection.
>> 6. once all evaluators have successfully checked in, driver will deem the
>> restart completed successfully.
>>
>> typing this off the top of my head... it should cover the outline of the
>> driver restart logic, there are other details and/or corner cases, for
>> example if evaluator(s) failed during driver's down time, the restart
>> logic
>> will be able to tell (some of the evaluators did not check back in), and
>> driver will fire FailedEvaluatorHandler on those evaluators. All cases
>> like
>> that are probably better illustrated by reading codes, i agree we could
>> have done better documentation though :)
>>
>>
>> markus was involved in the design and discussion during implementation, he
>> should be able to chip in as well.
>>
>> -yingda
>>
>>
>> On Mon, Jul 27, 2015 at 5:40 PM, Andrew Chung <af...@gmail.com>
>> wrote:
>>
>> > Hi Byung-Gon,
>> >
>> > As of now, there are still no architectural changes on driver
>> > restartability other than configuration changes.
>> >
>> > The process will be broken down into (at least) 3 steps:
>> > 1. Move restart configurations out of DriverConfiguration, since they
>> are
>> > not dependent on each other. Here, we will have a
>> > DriverRestartConfiguration as well as a
>> DriverRestartHandlersConfiguration.
>> > Parameters in DriverRestartConfiguration can only be bound once, while
>> > parameters in DriverRestartHandlersConfiguration can be bound multiple
>> > times, since restart handlers are all modified to be sets of handlers.
>> The
>> > RestartHandler in DriverRestartConfiguration is required, but the
>> > RestartHandler in DriverRestartHandlersConfiguration is not required,
>> since
>> > we need to guarantee at least one RestartHandler to be bound in order
>> for
>> > restarts to be functional.
>> > 2. Implement runtime specific evaluator preserving logic. I will only
>> > implement the YARN/HDInsight portion since I don't yet have experience
>> on
>> > Mesos. The new proposed way to preserve evaluators and retrieve them on
>> > restarted jobs is to have an injectable* runtime-specific*
>> > DriverRestartManager that handles detecting restarts and recording and
>> > retrieving evaluator IDs on restart. Currently, the functionality is
>> > completely embedded into YarnContainerManager, but I would like to make
>> the
>> > design more modular and configurable by delegating it out to another
>> > object. DriverRestartManager functions will be called by special
>> > ServiceEventHandlers that are bound in a restart configuration
>> mentioned in
>> > step 1.
>> > 3. Implement the actual restart architecture logic. I am not yet clear
>> on
>> > why the current restart mechanism for YARN is not working, but I will
>> get
>> > more details once steps 1 and 2 are done, and will create a
>> > proposal/diagram for it.
>> >
>> > The PR here:
>> > https://github.com/apache/incubator-reef/pull/319
>> > is for step 1.
>> >
>> > Please let me know if you would like clarification on anything.
>> >
>> > Thanks,
>> > Andrew
>> >
>> > On Mon, Jul 27, 2015 at 4:01 PM, Byung-Gon Chun <bg...@gmail.com>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > If there is a design doc on driver restartability, could you share
>> one?
>> > > It'd be helpful to have some diagram. (ref.
>> > > https://github.com/apache/incubator-reef/pull/310)
>> > >
>> > > Thanks.
>> > > -Gon
>> > >
>> > > On Tue, Jul 28, 2015 at 7:31 AM, Byung-Gon Chun <bg...@gmail.com>
>> > wrote:
>> > >
>> > > > Hi Andrew,
>> > > >
>> > > > Could you explain the rationale of your design a bit more in detail?
>> > > >
>> > > > Thanks.
>> > > > -Gon
>> > > >
>> > > > On Tue, Jul 28, 2015 at 3:32 AM, Andrew Chung <af...@gmail.com>
>> > > wrote:
>> > > >
>> > > >> Hi All,
>> > > >>
>> > > >> I will be changing DriverConfiguration by separating it into two
>> > > >> configuration modules:
>> > > >> 1. The runtime-dependent configuration module
>> > DriverRestartConfiguration
>> > > >> 2. The original configuration module DriverConfiguration, which
>> will
>> > not
>> > > >> have any restart parameters.
>> > > >>
>> > > >> In addition, there will be a change in the parameter
>> > > DriverRestartHandler,
>> > > >> which will change from Eventhandler<StartTime> to
>> > > >> Set<EventHandler<StartTime>>, and there will be a another new
>> > > >> ConfigurationModule, DriverRestartHandlersConfiguration, for driver
>> > > >> restarts to allow the users to bind the optional handlers multiple
>> > > times.
>> > > >> Note that the RestartHandler in DriverRestartConfiguration is
>> required
>> > > but
>> > > >> the RestartHandler in DriverRestartHandlersConfiguration will be
>> > > optional.
>> > > >>
>> > > >> Is this okay for everyone? I presume no one is using this feature
>> yet,
>> > > as
>> > > >> I
>> > > >> have tested it on an HDInsight cluster and confirmed that it
>> doesn't
>> > > work.
>> > > >>
>> > > >> Thanks,
>> > > >> Andrew
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Byung-Gon Chun
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Byung-Gon Chun
>> > >
>> >
>>
>
>
Re: Breaking backward compatibility in driver restart configurations
Posted by Andrew Chung <af...@gmail.com>.
Hi Yingda,
I think you may be right in that it works in the general case where only
the AM dies; however, when I tested the RM restart case with a modified
HelloREEF task with preserveContainersAcrossAppAttempts (modified the tasks
to be long-running), something didn't work correctly, and the driver
detected that all evaluators have turned idle and finished the application.
I have attached the logs on the restarted driver to this email. I can run a
few more tests with different variations, but there are probably some edge
cases that weren't covered before.
I do think it's still a good idea to delegate restart handlers and the
evaluator preservation logic to another configuration module though. What
do you think?
Thanks,
Andrew
On Mon, Jul 27, 2015 at 10:03 PM, Yingda Chen <yd...@gmail.com> wrote:
> the (old) driver HA works roughly with the following logic:
>
> 1. when driver HA is enabled, driver will record the come-and-go of every
> container on some hdfs file, so that at any time, there is record of how
> many containers are currently alive. in addition, driver will maintain a
> http server on which it keeps its connectivity information (ip and port
> etc.), the url is formed in a fixed format with application id.
> 2. assume that we have yarn support for container-retention switched
> on(default is off), when AM (driver) fails, yarn would not kill the
> containers associated with the AM, instead the containers will be kept
> alive.
> 3. the AM will be restarted after failure by RM (there should be a config
> for it too), and once it is restarted, it will look check with RM to see if
> there is any previous containers associated with it, if yes, it will know
> that it is reborn from fire and enter restart mode. at the same time, it
> will also start the http-server and put its connectivity information there.
> 4. for the Evaluators, when connection to driver is lost (after certain
> number of retires), evaluator will assume the driver has failed, and will
> then begin to ping the known driver http address (properly formatted with
> application id) periodically. once the ping goes through, it knows that the
> driver has restarted and can then get the ip and port information of the
> restarted driver. evaluator will then try to reconnect/register with the
> restarted driver
> 5. back to the (restarted) driver, it will be expecting evaluators to
> reconnect, and will do sanity check on every evaluator that tries to
> reconnect via container id (note that driver can get the list of expected
> evaluator ids from its previous life through RM). once an expected
> evaluator's reconnect request arrive, driver will do handshake and
> re-establish connection.
> 6. once all evaluators have successfully checked in, driver will deem the
> restart completed successfully.
>
> typing this off the top of my head... it should cover the outline of the
> driver restart logic, there are other details and/or corner cases, for
> example if evaluator(s) failed during driver's down time, the restart logic
> will be able to tell (some of the evaluators did not check back in), and
> driver will fire FailedEvaluatorHandler on those evaluators. All cases like
> that are probably better illustrated by reading codes, i agree we could
> have done better documentation though :)
>
>
> markus was involved in the design and discussion during implementation, he
> should be able to chip in as well.
>
> -yingda
>
>
> On Mon, Jul 27, 2015 at 5:40 PM, Andrew Chung <af...@gmail.com> wrote:
>
> > Hi Byung-Gon,
> >
> > As of now, there are still no architectural changes on driver
> > restartability other than configuration changes.
> >
> > The process will be broken down into (at least) 3 steps:
> > 1. Move restart configurations out of DriverConfiguration, since they are
> > not dependent on each other. Here, we will have a
> > DriverRestartConfiguration as well as a
> DriverRestartHandlersConfiguration.
> > Parameters in DriverRestartConfiguration can only be bound once, while
> > parameters in DriverRestartHandlersConfiguration can be bound multiple
> > times, since restart handlers are all modified to be sets of handlers.
> The
> > RestartHandler in DriverRestartConfiguration is required, but the
> > RestartHandler in DriverRestartHandlersConfiguration is not required,
> since
> > we need to guarantee at least one RestartHandler to be bound in order for
> > restarts to be functional.
> > 2. Implement runtime specific evaluator preserving logic. I will only
> > implement the YARN/HDInsight portion since I don't yet have experience on
> > Mesos. The new proposed way to preserve evaluators and retrieve them on
> > restarted jobs is to have an injectable* runtime-specific*
> > DriverRestartManager that handles detecting restarts and recording and
> > retrieving evaluator IDs on restart. Currently, the functionality is
> > completely embedded into YarnContainerManager, but I would like to make
> the
> > design more modular and configurable by delegating it out to another
> > object. DriverRestartManager functions will be called by special
> > ServiceEventHandlers that are bound in a restart configuration mentioned
> in
> > step 1.
> > 3. Implement the actual restart architecture logic. I am not yet clear on
> > why the current restart mechanism for YARN is not working, but I will get
> > more details once steps 1 and 2 are done, and will create a
> > proposal/diagram for it.
> >
> > The PR here:
> > https://github.com/apache/incubator-reef/pull/319
> > is for step 1.
> >
> > Please let me know if you would like clarification on anything.
> >
> > Thanks,
> > Andrew
> >
> > On Mon, Jul 27, 2015 at 4:01 PM, Byung-Gon Chun <bg...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > If there is a design doc on driver restartability, could you share one?
> > > It'd be helpful to have some diagram. (ref.
> > > https://github.com/apache/incubator-reef/pull/310)
> > >
> > > Thanks.
> > > -Gon
> > >
> > > On Tue, Jul 28, 2015 at 7:31 AM, Byung-Gon Chun <bg...@gmail.com>
> > wrote:
> > >
> > > > Hi Andrew,
> > > >
> > > > Could you explain the rationale of your design a bit more in detail?
> > > >
> > > > Thanks.
> > > > -Gon
> > > >
> > > > On Tue, Jul 28, 2015 at 3:32 AM, Andrew Chung <af...@gmail.com>
> > > wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> I will be changing DriverConfiguration by separating it into two
> > > >> configuration modules:
> > > >> 1. The runtime-dependent configuration module
> > DriverRestartConfiguration
> > > >> 2. The original configuration module DriverConfiguration, which will
> > not
> > > >> have any restart parameters.
> > > >>
> > > >> In addition, there will be a change in the parameter
> > > DriverRestartHandler,
> > > >> which will change from Eventhandler<StartTime> to
> > > >> Set<EventHandler<StartTime>>, and there will be a another new
> > > >> ConfigurationModule, DriverRestartHandlersConfiguration, for driver
> > > >> restarts to allow the users to bind the optional handlers multiple
> > > times.
> > > >> Note that the RestartHandler in DriverRestartConfiguration is
> required
> > > but
> > > >> the RestartHandler in DriverRestartHandlersConfiguration will be
> > > optional.
> > > >>
> > > >> Is this okay for everyone? I presume no one is using this feature
> yet,
> > > as
> > > >> I
> > > >> have tested it on an HDInsight cluster and confirmed that it doesn't
> > > work.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Byung-Gon Chun
> > > >
> > >
> > >
> > >
> > > --
> > > Byung-Gon Chun
> > >
> >
>
Re: Breaking backward compatibility in driver restart configurations
Posted by Yingda Chen <yd...@gmail.com>.
the (old) driver HA works roughly with the following logic:
1. when driver HA is enabled, driver will record the come-and-go of every
container on some hdfs file, so that at any time, there is record of how
many containers are currently alive. in addition, driver will maintain a
http server on which it keeps its connectivity information (ip and port
etc.), the url is formed in a fixed format with application id.
2. assume that we have yarn support for container-retention switched
on(default is off), when AM (driver) fails, yarn would not kill the
containers associated with the AM, instead the containers will be kept
alive.
3. the AM will be restarted after failure by RM (there should be a config
for it too), and once it is restarted, it will look check with RM to see if
there is any previous containers associated with it, if yes, it will know
that it is reborn from fire and enter restart mode. at the same time, it
will also start the http-server and put its connectivity information there.
4. for the Evaluators, when connection to driver is lost (after certain
number of retires), evaluator will assume the driver has failed, and will
then begin to ping the known driver http address (properly formatted with
application id) periodically. once the ping goes through, it knows that the
driver has restarted and can then get the ip and port information of the
restarted driver. evaluator will then try to reconnect/register with the
restarted driver
5. back to the (restarted) driver, it will be expecting evaluators to
reconnect, and will do sanity check on every evaluator that tries to
reconnect via container id (note that driver can get the list of expected
evaluator ids from its previous life through RM). once an expected
evaluator's reconnect request arrive, driver will do handshake and
re-establish connection.
6. once all evaluators have successfully checked in, driver will deem the
restart completed successfully.
typing this off the top of my head... it should cover the outline of the
driver restart logic, there are other details and/or corner cases, for
example if evaluator(s) failed during driver's down time, the restart logic
will be able to tell (some of the evaluators did not check back in), and
driver will fire FailedEvaluatorHandler on those evaluators. All cases like
that are probably better illustrated by reading codes, i agree we could
have done better documentation though :)
markus was involved in the design and discussion during implementation, he
should be able to chip in as well.
-yingda
On Mon, Jul 27, 2015 at 5:40 PM, Andrew Chung <af...@gmail.com> wrote:
> Hi Byung-Gon,
>
> As of now, there are still no architectural changes on driver
> restartability other than configuration changes.
>
> The process will be broken down into (at least) 3 steps:
> 1. Move restart configurations out of DriverConfiguration, since they are
> not dependent on each other. Here, we will have a
> DriverRestartConfiguration as well as a DriverRestartHandlersConfiguration.
> Parameters in DriverRestartConfiguration can only be bound once, while
> parameters in DriverRestartHandlersConfiguration can be bound multiple
> times, since restart handlers are all modified to be sets of handlers. The
> RestartHandler in DriverRestartConfiguration is required, but the
> RestartHandler in DriverRestartHandlersConfiguration is not required, since
> we need to guarantee at least one RestartHandler to be bound in order for
> restarts to be functional.
> 2. Implement runtime specific evaluator preserving logic. I will only
> implement the YARN/HDInsight portion since I don't yet have experience on
> Mesos. The new proposed way to preserve evaluators and retrieve them on
> restarted jobs is to have an injectable* runtime-specific*
> DriverRestartManager that handles detecting restarts and recording and
> retrieving evaluator IDs on restart. Currently, the functionality is
> completely embedded into YarnContainerManager, but I would like to make the
> design more modular and configurable by delegating it out to another
> object. DriverRestartManager functions will be called by special
> ServiceEventHandlers that are bound in a restart configuration mentioned in
> step 1.
> 3. Implement the actual restart architecture logic. I am not yet clear on
> why the current restart mechanism for YARN is not working, but I will get
> more details once steps 1 and 2 are done, and will create a
> proposal/diagram for it.
>
> The PR here:
> https://github.com/apache/incubator-reef/pull/319
> is for step 1.
>
> Please let me know if you would like clarification on anything.
>
> Thanks,
> Andrew
>
> On Mon, Jul 27, 2015 at 4:01 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
>
> > Hi,
> >
> > If there is a design doc on driver restartability, could you share one?
> > It'd be helpful to have some diagram. (ref.
> > https://github.com/apache/incubator-reef/pull/310)
> >
> > Thanks.
> > -Gon
> >
> > On Tue, Jul 28, 2015 at 7:31 AM, Byung-Gon Chun <bg...@gmail.com>
> wrote:
> >
> > > Hi Andrew,
> > >
> > > Could you explain the rationale of your design a bit more in detail?
> > >
> > > Thanks.
> > > -Gon
> > >
> > > On Tue, Jul 28, 2015 at 3:32 AM, Andrew Chung <af...@gmail.com>
> > wrote:
> > >
> > >> Hi All,
> > >>
> > >> I will be changing DriverConfiguration by separating it into two
> > >> configuration modules:
> > >> 1. The runtime-dependent configuration module
> DriverRestartConfiguration
> > >> 2. The original configuration module DriverConfiguration, which will
> not
> > >> have any restart parameters.
> > >>
> > >> In addition, there will be a change in the parameter
> > DriverRestartHandler,
> > >> which will change from Eventhandler<StartTime> to
> > >> Set<EventHandler<StartTime>>, and there will be a another new
> > >> ConfigurationModule, DriverRestartHandlersConfiguration, for driver
> > >> restarts to allow the users to bind the optional handlers multiple
> > times.
> > >> Note that the RestartHandler in DriverRestartConfiguration is required
> > but
> > >> the RestartHandler in DriverRestartHandlersConfiguration will be
> > optional.
> > >>
> > >> Is this okay for everyone? I presume no one is using this feature yet,
> > as
> > >> I
> > >> have tested it on an HDInsight cluster and confirmed that it doesn't
> > work.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >
> > >
> > >
> > > --
> > > Byung-Gon Chun
> > >
> >
> >
> >
> > --
> > Byung-Gon Chun
> >
>
Re: Breaking backward compatibility in driver restart configurations
Posted by Andrew Chung <af...@gmail.com>.
Hi Byung-Gon,
As of now, there are still no architectural changes on driver
restartability other than configuration changes.
The process will be broken down into (at least) 3 steps:
1. Move restart configurations out of DriverConfiguration, since they are
not dependent on each other. Here, we will have a
DriverRestartConfiguration as well as a DriverRestartHandlersConfiguration.
Parameters in DriverRestartConfiguration can only be bound once, while
parameters in DriverRestartHandlersConfiguration can be bound multiple
times, since restart handlers are all modified to be sets of handlers. The
RestartHandler in DriverRestartConfiguration is required, but the
RestartHandler in DriverRestartHandlersConfiguration is not required, since
we need to guarantee at least one RestartHandler to be bound in order for
restarts to be functional.
2. Implement runtime specific evaluator preserving logic. I will only
implement the YARN/HDInsight portion since I don't yet have experience on
Mesos. The new proposed way to preserve evaluators and retrieve them on
restarted jobs is to have an injectable* runtime-specific*
DriverRestartManager that handles detecting restarts and recording and
retrieving evaluator IDs on restart. Currently, the functionality is
completely embedded into YarnContainerManager, but I would like to make the
design more modular and configurable by delegating it out to another
object. DriverRestartManager functions will be called by special
ServiceEventHandlers that are bound in a restart configuration mentioned in
step 1.
3. Implement the actual restart architecture logic. I am not yet clear on
why the current restart mechanism for YARN is not working, but I will get
more details once steps 1 and 2 are done, and will create a
proposal/diagram for it.
The PR here:
https://github.com/apache/incubator-reef/pull/319
is for step 1.
Please let me know if you would like clarification on anything.
Thanks,
Andrew
On Mon, Jul 27, 2015 at 4:01 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
> Hi,
>
> If there is a design doc on driver restartability, could you share one?
> It'd be helpful to have some diagram. (ref.
> https://github.com/apache/incubator-reef/pull/310)
>
> Thanks.
> -Gon
>
> On Tue, Jul 28, 2015 at 7:31 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
>
> > Hi Andrew,
> >
> > Could you explain the rationale of your design a bit more in detail?
> >
> > Thanks.
> > -Gon
> >
> > On Tue, Jul 28, 2015 at 3:32 AM, Andrew Chung <af...@gmail.com>
> wrote:
> >
> >> Hi All,
> >>
> >> I will be changing DriverConfiguration by separating it into two
> >> configuration modules:
> >> 1. The runtime-dependent configuration module DriverRestartConfiguration
> >> 2. The original configuration module DriverConfiguration, which will not
> >> have any restart parameters.
> >>
> >> In addition, there will be a change in the parameter
> DriverRestartHandler,
> >> which will change from Eventhandler<StartTime> to
> >> Set<EventHandler<StartTime>>, and there will be a another new
> >> ConfigurationModule, DriverRestartHandlersConfiguration, for driver
> >> restarts to allow the users to bind the optional handlers multiple
> times.
> >> Note that the RestartHandler in DriverRestartConfiguration is required
> but
> >> the RestartHandler in DriverRestartHandlersConfiguration will be
> optional.
> >>
> >> Is this okay for everyone? I presume no one is using this feature yet,
> as
> >> I
> >> have tested it on an HDInsight cluster and confirmed that it doesn't
> work.
> >>
> >> Thanks,
> >> Andrew
> >>
> >
> >
> >
> > --
> > Byung-Gon Chun
> >
>
>
>
> --
> Byung-Gon Chun
>
Re: Breaking backward compatibility in driver restart configurations
Posted by Byung-Gon Chun <bg...@gmail.com>.
Hi,
If there is a design doc on driver restartability, could you share one?
It'd be helpful to have some diagram. (ref.
https://github.com/apache/incubator-reef/pull/310)
Thanks.
-Gon
On Tue, Jul 28, 2015 at 7:31 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
> Hi Andrew,
>
> Could you explain the rationale of your design a bit more in detail?
>
> Thanks.
> -Gon
>
> On Tue, Jul 28, 2015 at 3:32 AM, Andrew Chung <af...@gmail.com> wrote:
>
>> Hi All,
>>
>> I will be changing DriverConfiguration by separating it into two
>> configuration modules:
>> 1. The runtime-dependent configuration module DriverRestartConfiguration
>> 2. The original configuration module DriverConfiguration, which will not
>> have any restart parameters.
>>
>> In addition, there will be a change in the parameter DriverRestartHandler,
>> which will change from Eventhandler<StartTime> to
>> Set<EventHandler<StartTime>>, and there will be a another new
>> ConfigurationModule, DriverRestartHandlersConfiguration, for driver
>> restarts to allow the users to bind the optional handlers multiple times.
>> Note that the RestartHandler in DriverRestartConfiguration is required but
>> the RestartHandler in DriverRestartHandlersConfiguration will be optional.
>>
>> Is this okay for everyone? I presume no one is using this feature yet, as
>> I
>> have tested it on an HDInsight cluster and confirmed that it doesn't work.
>>
>> Thanks,
>> Andrew
>>
>
>
>
> --
> Byung-Gon Chun
>
--
Byung-Gon Chun
Re: Breaking backward compatibility in driver restart configurations
Posted by Byung-Gon Chun <bg...@gmail.com>.
Hi Andrew,
Could you explain the rationale of your design a bit more in detail?
Thanks.
-Gon
On Tue, Jul 28, 2015 at 3:32 AM, Andrew Chung <af...@gmail.com> wrote:
> Hi All,
>
> I will be changing DriverConfiguration by separating it into two
> configuration modules:
> 1. The runtime-dependent configuration module DriverRestartConfiguration
> 2. The original configuration module DriverConfiguration, which will not
> have any restart parameters.
>
> In addition, there will be a change in the parameter DriverRestartHandler,
> which will change from Eventhandler<StartTime> to
> Set<EventHandler<StartTime>>, and there will be a another new
> ConfigurationModule, DriverRestartHandlersConfiguration, for driver
> restarts to allow the users to bind the optional handlers multiple times.
> Note that the RestartHandler in DriverRestartConfiguration is required but
> the RestartHandler in DriverRestartHandlersConfiguration will be optional.
>
> Is this okay for everyone? I presume no one is using this feature yet, as I
> have tested it on an HDInsight cluster and confirmed that it doesn't work.
>
> Thanks,
> Andrew
>
--
Byung-Gon Chun
Re: Breaking backward compatibility in driver restart configurations
Posted by Yingda Chen <yd...@gmail.com>.
do you have the yarn config to retain containers after AM fails set to true?
unfortunately there was no automatic test since the restartbility was not
supported in local runtime, it was tested on both Yarn one-box installation
and actual HDInsight cluster with hadoop 2.4.1.
On Mon, Jul 27, 2015 at 12:50 PM, Andrew Chung <af...@gmail.com> wrote:
> From what I observed, the evaluators turned idle for our ASA job and the
> job finished after restart. Did you have a test in the codebase?
>
> Thanks,
> Andrew
>
> On Mon, Jul 27, 2015 at 12:09 PM, Yingda Chen <yd...@gmail.com> wrote:
>
> > pretty sure it was driver restart was working on hdp2.4.1+ cluster when
> it
> > was first implemented...
> >
> > On Mon, Jul 27, 2015 at 11:32 AM, Andrew Chung <af...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I will be changing DriverConfiguration by separating it into two
> > > configuration modules:
> > > 1. The runtime-dependent configuration module
> DriverRestartConfiguration
> > > 2. The original configuration module DriverConfiguration, which will
> not
> > > have any restart parameters.
> > >
> > > In addition, there will be a change in the parameter
> > DriverRestartHandler,
> > > which will change from Eventhandler<StartTime> to
> > > Set<EventHandler<StartTime>>, and there will be a another new
> > > ConfigurationModule, DriverRestartHandlersConfiguration, for driver
> > > restarts to allow the users to bind the optional handlers multiple
> times.
> > > Note that the RestartHandler in DriverRestartConfiguration is required
> > but
> > > the RestartHandler in DriverRestartHandlersConfiguration will be
> > optional.
> > >
> > > Is this okay for everyone? I presume no one is using this feature yet,
> > as I
> > > have tested it on an HDInsight cluster and confirmed that it doesn't
> > work.
> > >
> > > Thanks,
> > > Andrew
> > >
> >
>
Re: Breaking backward compatibility in driver restart configurations
Posted by Andrew Chung <af...@gmail.com>.
>From what I observed, the evaluators turned idle for our ASA job and the
job finished after restart. Did you have a test in the codebase?
Thanks,
Andrew
On Mon, Jul 27, 2015 at 12:09 PM, Yingda Chen <yd...@gmail.com> wrote:
> pretty sure it was driver restart was working on hdp2.4.1+ cluster when it
> was first implemented...
>
> On Mon, Jul 27, 2015 at 11:32 AM, Andrew Chung <af...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I will be changing DriverConfiguration by separating it into two
> > configuration modules:
> > 1. The runtime-dependent configuration module DriverRestartConfiguration
> > 2. The original configuration module DriverConfiguration, which will not
> > have any restart parameters.
> >
> > In addition, there will be a change in the parameter
> DriverRestartHandler,
> > which will change from Eventhandler<StartTime> to
> > Set<EventHandler<StartTime>>, and there will be a another new
> > ConfigurationModule, DriverRestartHandlersConfiguration, for driver
> > restarts to allow the users to bind the optional handlers multiple times.
> > Note that the RestartHandler in DriverRestartConfiguration is required
> but
> > the RestartHandler in DriverRestartHandlersConfiguration will be
> optional.
> >
> > Is this okay for everyone? I presume no one is using this feature yet,
> as I
> > have tested it on an HDInsight cluster and confirmed that it doesn't
> work.
> >
> > Thanks,
> > Andrew
> >
>
Re: Breaking backward compatibility in driver restart configurations
Posted by Yingda Chen <yd...@gmail.com>.
pretty sure it was driver restart was working on hdp2.4.1+ cluster when it
was first implemented...
On Mon, Jul 27, 2015 at 11:32 AM, Andrew Chung <af...@gmail.com> wrote:
> Hi All,
>
> I will be changing DriverConfiguration by separating it into two
> configuration modules:
> 1. The runtime-dependent configuration module DriverRestartConfiguration
> 2. The original configuration module DriverConfiguration, which will not
> have any restart parameters.
>
> In addition, there will be a change in the parameter DriverRestartHandler,
> which will change from Eventhandler<StartTime> to
> Set<EventHandler<StartTime>>, and there will be a another new
> ConfigurationModule, DriverRestartHandlersConfiguration, for driver
> restarts to allow the users to bind the optional handlers multiple times.
> Note that the RestartHandler in DriverRestartConfiguration is required but
> the RestartHandler in DriverRestartHandlersConfiguration will be optional.
>
> Is this okay for everyone? I presume no one is using this feature yet, as I
> have tested it on an HDInsight cluster and confirmed that it doesn't work.
>
> Thanks,
> Andrew
>