You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Jake Maes <ja...@gmail.com> on 2016/03/22 01:43:29 UTC
Re: Review Request 45136: SAMZA-905 Host Affinity - State restore
doesn't work if the previous shutdown was uncontrolled
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45136/
-----------------------------------------------------------
(Updated March 22, 2016, 12:43 a.m.)
Review request for samza, Navina Ramesh, Jagadish Venkatraman, and Yi Pan (Data Infrastructure).
Summary (updated)
-----------------
SAMZA-905 Host Affinity - State restore doesn't work if the previous shutdown was uncontrolled
Repository: samza
Description
-------
Add continuous checkpointing.
The offset file is now written on each commit
Diffs
-----
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 422799a099dd391a50f934fe65d5b37471264af6
samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala f08e8ddd6fc3b2cbf4ca195f22049d368abe6e62
Diff: https://reviews.apache.org/r/45136/diff/
Testing
-------
New unit tests and a test job. With the test job set to autocommit, I was able to see the OFFSET file get written every commit.
Thanks,
Jake Maes
Re: Review Request 45136: SAMZA-905 Host Affinity - State restore
doesn't work if the previous shutdown was uncontrolled
Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45136/#review125332
-----------------------------------------------------------
Ship it!
Sorry for the late review. Changes look good. +1
- Navina Ramesh
On March 24, 2016, 12:30 a.m., Jake Maes wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45136/
> -----------------------------------------------------------
>
> (Updated March 24, 2016, 12:30 a.m.)
>
>
> Review request for samza, Navina Ramesh, Jagadish Venkatraman, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Add continuous checkpointing.
> The offset file is now written on each commit
>
>
> Diffs
> -----
>
> docs/learn/documentation/versioned/yarn/yarn-host-affinity.md 1d9c29e7ee7fd19fabba55111b5088cc5d36f1fe
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 422799a099dd391a50f934fe65d5b37471264af6
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala f08e8ddd6fc3b2cbf4ca195f22049d368abe6e62
>
> Diff: https://reviews.apache.org/r/45136/diff/
>
>
> Testing
> -------
>
> New unit tests and a test job. With the test job set to autocommit, I was able to see the OFFSET file get written every commit.
>
>
> Thanks,
>
> Jake Maes
>
>
Re: Review Request 45136: SAMZA-905 Host Affinity - State restore
doesn't work if the previous shutdown was uncontrolled
Posted by Jake Maes <ja...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45136/
-----------------------------------------------------------
(Updated March 24, 2016, 12:30 a.m.)
Review request for samza, Navina Ramesh, Jagadish Venkatraman, and Yi Pan (Data Infrastructure).
Repository: samza
Description
-------
Add continuous checkpointing.
The offset file is now written on each commit
Diffs (updated)
-----
docs/learn/documentation/versioned/yarn/yarn-host-affinity.md 1d9c29e7ee7fd19fabba55111b5088cc5d36f1fe
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 422799a099dd391a50f934fe65d5b37471264af6
samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala f08e8ddd6fc3b2cbf4ca195f22049d368abe6e62
Diff: https://reviews.apache.org/r/45136/diff/
Testing
-------
New unit tests and a test job. With the test job set to autocommit, I was able to see the OFFSET file get written every commit.
Thanks,
Jake Maes
Re: Review Request 45136: SAMZA-905 Host Affinity - State restore
doesn't work if the previous shutdown was uncontrolled
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45136/#review125146
-----------------------------------------------------------
Fix it, then Ship it!
Fix the document and +1.
docs/learn/documentation/versioned/yarn/yarn-host-affinity.md (line 24)
<https://reviews.apache.org/r/45136/#comment187912>
The mapping between task instance and the data stores is not necessary 1:1. There could be multiple data stores the user uses in a single task instance, hence 1:N
- Yi Pan (Data Infrastructure)
On March 22, 2016, 9:27 p.m., Jake Maes wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45136/
> -----------------------------------------------------------
>
> (Updated March 22, 2016, 9:27 p.m.)
>
>
> Review request for samza, Navina Ramesh, Jagadish Venkatraman, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> Add continuous checkpointing.
> The offset file is now written on each commit
>
>
> Diffs
> -----
>
> docs/learn/documentation/versioned/yarn/yarn-host-affinity.md 1d9c29e7ee7fd19fabba55111b5088cc5d36f1fe
> samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 422799a099dd391a50f934fe65d5b37471264af6
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala f08e8ddd6fc3b2cbf4ca195f22049d368abe6e62
>
> Diff: https://reviews.apache.org/r/45136/diff/
>
>
> Testing
> -------
>
> New unit tests and a test job. With the test job set to autocommit, I was able to see the OFFSET file get written every commit.
>
>
> Thanks,
>
> Jake Maes
>
>
Re: Review Request 45136: SAMZA-905 Host Affinity - State restore
doesn't work if the previous shutdown was uncontrolled
Posted by Jake Maes <ja...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45136/
-----------------------------------------------------------
(Updated March 22, 2016, 9:27 p.m.)
Review request for samza, Navina Ramesh, Jagadish Venkatraman, and Yi Pan (Data Infrastructure).
Changes
-------
Adding doc updates
Repository: samza
Description
-------
Add continuous checkpointing.
The offset file is now written on each commit
Diffs (updated)
-----
docs/learn/documentation/versioned/yarn/yarn-host-affinity.md 1d9c29e7ee7fd19fabba55111b5088cc5d36f1fe
samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 422799a099dd391a50f934fe65d5b37471264af6
samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala f08e8ddd6fc3b2cbf4ca195f22049d368abe6e62
Diff: https://reviews.apache.org/r/45136/diff/
Testing
-------
New unit tests and a test job. With the test job set to autocommit, I was able to see the OFFSET file get written every commit.
Thanks,
Jake Maes