You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by David Yu <da...@optimizely.com> on 2016/08/24 22:39:55 UTC
Job coordinator stream and job redeployment
Hi,
I'm trying to understand role of the coordinator stream during a job
redeployment.
From the Samza documentation, I'm seeing the following about the
coordinator stream:
The Job Coordinator bootstraps configuration from the coordinator stream
each time upon job start-up. It periodically catches up with any new data
written to the coordinator stream and updates the Job Model.
However, it is unclear to me how this will work with "--config-path" when
we need to redeploy a job. Does the provided config first gets persisted to
the coordinator stream, updating the previous model, or will it simply be
ignored?
Thanks,
David
Re: Job coordinator stream and job redeployment
Posted by David Yu <da...@optimizely.com>.
Thanks for confirming, Jake :)
On Thu, Aug 25, 2016 at 11:18 AM Jacob Maes <ja...@gmail.com> wrote:
> Hey David,
>
> Yes, the config is rewritten by the JobRunner each time the job is
> restarted as dictated by the "resetJobConfig" property (which defaults to
> true) here:
>
> https://github.com/apache/samza/blob/944dd02e1d00bcce59f1fcc33ecbb2a8acd95870/samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala#L85
>
> The version parameter is not the version of the config, but the version of
> the coordinator stream protocol. It will only increment if there's a change
> to the way Samza reads/writes coordinator stream messages.
>
> -Jake
>
> On Thu, Aug 25, 2016 at 8:44 AM, David Yu <da...@optimizely.com> wrote:
>
> > After digging around a bit using kafka-console-consumer.sh, I'm able to
> > peek into the coordinator stream and see the config entries.
> >
> > Looks like a redeployment would just append the new configs to this
> topic.
> > And I assume the bootstrapping process will have to consume the entire
> > stream from the beginning to get a consolidated/up-to-date job model. Am
> I
> > understanding this correctly?
> >
> > Also, according to the doc, the keys of the coordinator stream is
> > ["<version-number>", "<message-type>", "<key>"]. From what I saw, the
> > <version-number> doesn't seem to change. It stays at 1 even after I
> > redeployed the job with new configs. I was assuming that it would get
> > incremented with every deployment/update.
> >
> > Thanks,
> > David
> >
> > On Wed, Aug 24, 2016 at 5:39 PM David Yu <da...@optimizely.com>
> wrote:
> >
> > > Hi,
> > >
> > > I'm trying to understand role of the coordinator stream during a job
> > > redeployment.
> > >
> > > From the Samza documentation, I'm seeing the following about the
> > > coordinator stream:
> > >
> > > The Job Coordinator bootstraps configuration from the coordinator
> stream
> > > each time upon job start-up. It periodically catches up with any new
> data
> > > written to the coordinator stream and updates the Job Model.
> > >
> > > However, it is unclear to me how this will work with "--config-path"
> when
> > > we need to redeploy a job. Does the provided config first gets
> persisted
> > to
> > > the coordinator stream, updating the previous model, or will it simply
> be
> > > ignored?
> > >
> > > Thanks,
> > > David
> > >
> >
>
Re: Job coordinator stream and job redeployment
Posted by Jacob Maes <ja...@gmail.com>.
Hey David,
Yes, the config is rewritten by the JobRunner each time the job is
restarted as dictated by the "resetJobConfig" property (which defaults to
true) here:
https://github.com/apache/samza/blob/944dd02e1d00bcce59f1fcc33ecbb2a8acd95870/samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala#L85
The version parameter is not the version of the config, but the version of
the coordinator stream protocol. It will only increment if there's a change
to the way Samza reads/writes coordinator stream messages.
-Jake
On Thu, Aug 25, 2016 at 8:44 AM, David Yu <da...@optimizely.com> wrote:
> After digging around a bit using kafka-console-consumer.sh, I'm able to
> peek into the coordinator stream and see the config entries.
>
> Looks like a redeployment would just append the new configs to this topic.
> And I assume the bootstrapping process will have to consume the entire
> stream from the beginning to get a consolidated/up-to-date job model. Am I
> understanding this correctly?
>
> Also, according to the doc, the keys of the coordinator stream is
> ["<version-number>", "<message-type>", "<key>"]. From what I saw, the
> <version-number> doesn't seem to change. It stays at 1 even after I
> redeployed the job with new configs. I was assuming that it would get
> incremented with every deployment/update.
>
> Thanks,
> David
>
> On Wed, Aug 24, 2016 at 5:39 PM David Yu <da...@optimizely.com> wrote:
>
> > Hi,
> >
> > I'm trying to understand role of the coordinator stream during a job
> > redeployment.
> >
> > From the Samza documentation, I'm seeing the following about the
> > coordinator stream:
> >
> > The Job Coordinator bootstraps configuration from the coordinator stream
> > each time upon job start-up. It periodically catches up with any new data
> > written to the coordinator stream and updates the Job Model.
> >
> > However, it is unclear to me how this will work with "--config-path" when
> > we need to redeploy a job. Does the provided config first gets persisted
> to
> > the coordinator stream, updating the previous model, or will it simply be
> > ignored?
> >
> > Thanks,
> > David
> >
>
Re: Job coordinator stream and job redeployment
Posted by David Yu <da...@optimizely.com>.
After digging around a bit using kafka-console-consumer.sh, I'm able to
peek into the coordinator stream and see the config entries.
Looks like a redeployment would just append the new configs to this topic.
And I assume the bootstrapping process will have to consume the entire
stream from the beginning to get a consolidated/up-to-date job model. Am I
understanding this correctly?
Also, according to the doc, the keys of the coordinator stream is
["<version-number>", "<message-type>", "<key>"]. From what I saw, the
<version-number> doesn't seem to change. It stays at 1 even after I
redeployed the job with new configs. I was assuming that it would get
incremented with every deployment/update.
Thanks,
David
On Wed, Aug 24, 2016 at 5:39 PM David Yu <da...@optimizely.com> wrote:
> Hi,
>
> I'm trying to understand role of the coordinator stream during a job
> redeployment.
>
> From the Samza documentation, I'm seeing the following about the
> coordinator stream:
>
> The Job Coordinator bootstraps configuration from the coordinator stream
> each time upon job start-up. It periodically catches up with any new data
> written to the coordinator stream and updates the Job Model.
>
> However, it is unclear to me how this will work with "--config-path" when
> we need to redeploy a job. Does the provided config first gets persisted to
> the coordinator stream, updating the previous model, or will it simply be
> ignored?
>
> Thanks,
> David
>