You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Martin Kleppmann <mk...@linkedin.com> on 2014/05/02 00:14:12 UTC
Review Request 20989: SAMZA-252: Add page on reprocessing to the docs
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20989/
-----------------------------------------------------------
Review request for samza.
Repository: samza
Description
-------
SAMZA-252: Add page on reprocessing to the docs.
Diffs
-----
docs/_layouts/default.html 0a5ad9f63110c68424360773b9fcd005e4a059a9
docs/learn/documentation/0.7.0/index.html 7806baf71bee61e5316d5bc627fee219012d3375
docs/learn/documentation/0.7.0/jobs/logging.md 6bb6bf4b3630165159acc47e4cfb8e1afe6659cb
docs/learn/documentation/0.7.0/jobs/reprocessing.md PRE-CREATION
Diff: https://reviews.apache.org/r/20989/diff/
Testing
-------
Thanks,
Martin Kleppmann
Re: Review Request 20989: SAMZA-252: Add page on reprocessing to the docs
Posted by Jay Kreps <bo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20989/#review42107
-----------------------------------------------------------
docs/learn/documentation/0.7.0/jobs/reprocessing.md
<https://reviews.apache.org/r/20989/#comment75851>
I think it would be more clear here to say "From time to time you may want to deploy a new version of your Samza job that computes results differently. Perhaps you fixed a bug or introduced a new feature.
docs/learn/documentation/0.7.0/jobs/reprocessing.md
<https://reviews.apache.org/r/20989/#comment75852>
Worth mentioning that you can increase your container count to make this as fast as you like.
docs/learn/documentation/0.7.0/jobs/reprocessing.md
<https://reviews.apache.org/r/20989/#comment75853>
I would actually argue for documenting the two topic solution: original job publishes to A and new job publishes to A'. This solution is generally correct. I think you can then mention making A = A' as a special case if the consumers can handle that.
- Jay Kreps
On May 1, 2014, 10:14 p.m., Martin Kleppmann wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20989/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 10:14 p.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA-252: Add page on reprocessing to the docs.
>
>
> Diffs
> -----
>
> docs/_layouts/default.html 0a5ad9f63110c68424360773b9fcd005e4a059a9
> docs/learn/documentation/0.7.0/index.html 7806baf71bee61e5316d5bc627fee219012d3375
> docs/learn/documentation/0.7.0/jobs/logging.md 6bb6bf4b3630165159acc47e4cfb8e1afe6659cb
> docs/learn/documentation/0.7.0/jobs/reprocessing.md PRE-CREATION
>
> Diff: https://reviews.apache.org/r/20989/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Martin Kleppmann
>
>
Re: Review Request 20989: SAMZA-252: Add page on reprocessing to the docs
Posted by Jay Kreps <bo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20989/#review42108
-----------------------------------------------------------
This is good. As a high-level comment I think it might be good to introduce this topic by saying that there are several ways to handle reprocessing, give each a name, and then discuss pros and cons and document the checkpoint stuff. I think it would be worth covering:
1. "Simple rewind": Just delete your state and change the job checkpoint to 0. (pro: super simple, con: some downtime).
2. "Parallel rewind": Restart a second copy of the job going to a new output topic in parallel. Switch consumers when caught up. Alternately can share output topics if the consumers can handle that. (pro: still pretty simple, con: need to change consumer too).
3. Lambda architecture: Reimplement job in Hadoop and do the reprocessing there (downside: need to somehow maintain logic and operate in two systems).
- Jay Kreps
On May 1, 2014, 10:14 p.m., Martin Kleppmann wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20989/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 10:14 p.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA-252: Add page on reprocessing to the docs.
>
>
> Diffs
> -----
>
> docs/_layouts/default.html 0a5ad9f63110c68424360773b9fcd005e4a059a9
> docs/learn/documentation/0.7.0/index.html 7806baf71bee61e5316d5bc627fee219012d3375
> docs/learn/documentation/0.7.0/jobs/logging.md 6bb6bf4b3630165159acc47e4cfb8e1afe6659cb
> docs/learn/documentation/0.7.0/jobs/reprocessing.md PRE-CREATION
>
> Diff: https://reviews.apache.org/r/20989/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Martin Kleppmann
>
>
Re: Review Request 20989: SAMZA-252: Add page on reprocessing to the docs
Posted by Martin Kleppmann <mk...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20989/
-----------------------------------------------------------
(Updated June 13, 2014, 5:27 p.m.)
Review request for samza.
Changes
-------
Thanks for the helpful feedback! Reworked the page based on review feedback. Also rebased onto master.
Repository: samza
Description
-------
SAMZA-252: Add page on reprocessing to the docs.
Diffs (updated)
-----
docs/_config.yml 2b90b08bafc5746efeae486daebc458f5c41847d
docs/learn/documentation/0.7.0/container/checkpointing.md 5b5261d998aed01530321e285a03115791cfba6d
docs/learn/documentation/0.7.0/index.html 695abf235b0bbdb9f7d8e351f8c04d3b796c976a
docs/learn/documentation/0.7.0/jobs/logging.md 9b5fea87bde15a6f034380c2dd4324d1127a8e93
docs/learn/documentation/0.7.0/jobs/reprocessing.md PRE-CREATION
Diff: https://reviews.apache.org/r/20989/diff/
Testing
-------
Thanks,
Martin Kleppmann
Re: Review Request 20989: SAMZA-252: Add page on reprocessing to the docs
Posted by Martin Kleppmann <mk...@linkedin.com>.
> On May 5, 2014, 4:58 p.m., Chris Riccomini wrote:
> > docs/learn/documentation/0.7.0/jobs/reprocessing.md, line 10
> > <https://reviews.apache.org/r/20989/diff/1/?file=573070#file573070line10>
> >
> > "If the new version produces different output from the old version, anyone who is consuming your job's output will see the output stream switch from the old version to the new version at this point."
> >
> > This is a little confusing. I associate the word "output" with the payload of a message. The word "stream" I associate with an output stream. If a job produces different output, that to me sounds like it's adding a field, but sending to the same stream. I think you're actually saying if the job produces to a different stream in the newer version, right?
Yes, that's what I meant, but it's confusing. I'll just delete that sentence, as I think it's more confusing than helpful.
> On May 5, 2014, 4:58 p.m., Chris Riccomini wrote:
> > docs/learn/documentation/0.7.0/jobs/reprocessing.md, line 57
> > <https://reviews.apache.org/r/20989/diff/1/?file=573070#file573070line57>
> >
> > Might be useful to provide a real-world example here, just to make things clear. This concept is a bit abstract without detailed explanation.
Good point, I've added one.
- Martin
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20989/#review42161
-----------------------------------------------------------
On June 13, 2014, 5:27 p.m., Martin Kleppmann wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20989/
> -----------------------------------------------------------
>
> (Updated June 13, 2014, 5:27 p.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA-252: Add page on reprocessing to the docs.
>
>
> Diffs
> -----
>
> docs/_config.yml 2b90b08bafc5746efeae486daebc458f5c41847d
> docs/learn/documentation/0.7.0/container/checkpointing.md 5b5261d998aed01530321e285a03115791cfba6d
> docs/learn/documentation/0.7.0/index.html 695abf235b0bbdb9f7d8e351f8c04d3b796c976a
> docs/learn/documentation/0.7.0/jobs/logging.md 9b5fea87bde15a6f034380c2dd4324d1127a8e93
> docs/learn/documentation/0.7.0/jobs/reprocessing.md PRE-CREATION
>
> Diff: https://reviews.apache.org/r/20989/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Martin Kleppmann
>
>
Re: Review Request 20989: SAMZA-252: Add page on reprocessing to the docs
Posted by Chris Riccomini <cr...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20989/#review42161
-----------------------------------------------------------
docs/learn/documentation/0.7.0/jobs/reprocessing.md
<https://reviews.apache.org/r/20989/#comment75879>
"If the new version produces different output from the old version, anyone who is consuming your job's output will see the output stream switch from the old version to the new version at this point."
This is a little confusing. I associate the word "output" with the payload of a message. The word "stream" I associate with an output stream. If a job produces different output, that to me sounds like it's adding a field, but sending to the same stream. I think you're actually saying if the job produces to a different stream in the newer version, right?
docs/learn/documentation/0.7.0/jobs/reprocessing.md
<https://reviews.apache.org/r/20989/#comment75880>
Agree.
docs/learn/documentation/0.7.0/jobs/reprocessing.md
<https://reviews.apache.org/r/20989/#comment75881>
Might be useful to provide a real-world example here, just to make things clear. This concept is a bit abstract without detailed explanation.
- Chris Riccomini
On May 1, 2014, 10:14 p.m., Martin Kleppmann wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20989/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 10:14 p.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA-252: Add page on reprocessing to the docs.
>
>
> Diffs
> -----
>
> docs/_layouts/default.html 0a5ad9f63110c68424360773b9fcd005e4a059a9
> docs/learn/documentation/0.7.0/index.html 7806baf71bee61e5316d5bc627fee219012d3375
> docs/learn/documentation/0.7.0/jobs/logging.md 6bb6bf4b3630165159acc47e4cfb8e1afe6659cb
> docs/learn/documentation/0.7.0/jobs/reprocessing.md PRE-CREATION
>
> Diff: https://reviews.apache.org/r/20989/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Martin Kleppmann
>
>