You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Naveen Somasundaram <na...@gmail.com> on 2015/04/01 19:37:16 UTC
Re: Review Request 32147: SAMZA-465
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/
-----------------------------------------------------------
(Updated April 1, 2015, 5:37 p.m.)
Review request for samza.
Repository: samza
Description
-------
SAMZA 465
Diffs (updated)
-----
build.gradle 0a268ac7a3819cf46b54a93e0e3171455371456a
samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 275eb1a924d09a0a43efe6273e0d2af9217e1c74
samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 81742bc93bcbca5a2ed43c701bcd5d3f40d79bfe
samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala 4a1b31f025ba7b05a7b46041aa8e12074599ce24
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 7fc6d89b5d703a7c10a212aaa8d3f9231996b897
samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala b790be17cfe08da28220ffb381cbd618ebe25cf0
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4f15002325bc0154991f9a35312e903d15ef81e7
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 3d1e6ecbb3fd95816c722a68c4f5907120eb20d0
samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java 4ef3551f470e77e27bd156e81ce96486f25c21bf
samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala a8b724bf781003142e455fdf1fed2f13d6c18353
samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
Diff: https://reviews.apache.org/r/32147/diff/
Testing
-------
The patch does the following:
1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
3. Passes the checkpoint information to the containers from the jobcoordinator
4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
Tests:
All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
Pending Issues:
1. The metrics registry is right now not used correctly, I need to pass the right reference to it
2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
Thanks,
Naveen Somasundaram
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
> On April 2, 2015, 11:35 p.m., Chris Riccomini wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java, line 458
> > <https://reviews.apache.org/r/32147/diff/2-5/?file=897282#file897282line458>
> >
> > typo
fixed
> On April 2, 2015, 11:35 p.m., Chris Riccomini wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, line 177
> > <https://reviews.apache.org/r/32147/diff/2-5/?file=897283#file897283line177>
> >
> > nit: () {
> >
> > single line.
fixed.
> On April 2, 2015, 11:35 p.m., Chris Riccomini wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, line 182
> > <https://reviews.apache.org/r/32147/diff/2-5/?file=897283#file897283line182>
> >
> > Might want to do type.equalsIgnoreCase(coordinatorStreamMessage.getType()) in the off chance that a message with a null type (malformed) gets into the stream.
> >
> > Or maybe check for nulls and throw a more meaningful error than an NPE.
Good catch. Fixed.
> On April 2, 2015, 11:35 p.m., Chris Riccomini wrote:
> > samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala, line 32
> > <https://reviews.apache.org/r/32147/diff/2-5/?file=897297#file897297line32>
> >
> > Delete
done
> On April 2, 2015, 11:35 p.m., Chris Riccomini wrote:
> > samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java, line 185
> > <https://reviews.apache.org/r/32147/diff/2-5/?file=897331#file897331line185>
> >
> > Does it make sense to just have JobCoordinator.apply default to new MetricsRegistryMap? It seems like we had to make a bunch of changes rather than just set the default in the constructor, and leave the code alone everywhere.
Because of java vs scala. Java doesn't undestand default constructor args. I have added two constructors, looks better now.
- Naveen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review78743
-----------------------------------------------------------
On April 1, 2015, 9:02 p.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 1, 2015, 9:02 p.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 87fcf589b10fd952df630d498c645159ed381739
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 81742bc93bcbca5a2ed43c701bcd5d3f40d79bfe
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 7fc6d89b5d703a7c10a212aaa8d3f9231996b897
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4f15002325bc0154991f9a35312e903d15ef81e7
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Chris Riccomini <cr...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review78743
-----------------------------------------------------------
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java
<https://reviews.apache.org/r/32147/#comment127739>
Chould this be container ID?
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java
<https://reviews.apache.org/r/32147/#comment127740>
Chould this be container ID?
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java
<https://reviews.apache.org/r/32147/#comment127741>
typo
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
<https://reviews.apache.org/r/32147/#comment127738>
nit: () {
single line.
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
<https://reviews.apache.org/r/32147/#comment127742>
Might want to do type.equalsIgnoreCase(coordinatorStreamMessage.getType()) in the off chance that a message with a null type (malformed) gets into the stream.
Or maybe check for nulls and throw a more meaningful error than an NPE.
samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala
<https://reviews.apache.org/r/32147/#comment127744>
nit: lowercase j
samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala
<https://reviews.apache.org/r/32147/#comment127745>
Delete
samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java
<https://reviews.apache.org/r/32147/#comment127747>
Does it make sense to just have JobCoordinator.apply default to new MetricsRegistryMap? It seems like we had to make a bunch of changes rather than just set the default in the constructor, and leave the code alone everywhere.
- Chris Riccomini
On April 1, 2015, 9:02 p.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 1, 2015, 9:02 p.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 87fcf589b10fd952df630d498c645159ed381739
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 81742bc93bcbca5a2ed43c701bcd5d3f40d79bfe
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 7fc6d89b5d703a7c10a212aaa8d3f9231996b897
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4f15002325bc0154991f9a35312e903d15ef81e7
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Yan Fang <ya...@gmail.com>.
> On April 29, 2015, 8:38 p.m., Naveen Somasundaram wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java, line 448
> > <https://reviews.apache.org/r/32147/diff/7/?file=922533#file922533line448>
> >
> > This is consistent with how it was previously in KafkaCheckpointManager. The partition is per task, but the topic is not the same. Each of the task will use the same Partition, but with different changelog topic name for each store.
> >
> > Relevant part of code from SamzaContainer:
> > ```
> > val changeLogSystemStreams = config
> > .getStoreNames
> > .filter(config.getChangelogStream(_).isDefined)
> > .map(name => (name, config.getChangelogStream(name).get)).toMap
> > .mapValues(Util.getSystemStreamFromNames(_))
> > ```
> >
> > And each of the store stream will use the same partition, which is fetched from this mapping.
Ok. I see. You mean we are not using the coordinatorStream for the changelog, they still are using the seperate streams, right?
> On April 29, 2015, 8:38 p.m., Naveen Somasundaram wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, line 60
> > <https://reviews.apache.org/r/32147/diff/7/?file=922534#file922534line60>
> >
> > boolean default in java is false. Or is there something else you are referring to ?
that's fine. I mean this. just sometimes not very explicit.
- Yan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review82006
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review82006
-----------------------------------------------------------
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java
<https://reviews.apache.org/r/32147/#comment132584>
We need to bootstrap everytime because, if the container fails and comes back up, then we need to fetch the latest checkpoint from the coordinator stream before we serve. This logic happens in JobCoordinator, for every HTTP request, we refresh the JobModel, which ensures we always have the latest JobModel (which has the latest checkpoint, because we bootstrap everytime).
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java
<https://reviews.apache.org/r/32147/#comment132594>
This is consistent with how it was previously in KafkaCheckpointManager. The partition is per task, but the topic is not the same. Each of the task will use the same Partition, but with different changelog topic name for each store.
Relevant part of code from SamzaContainer:
```
val changeLogSystemStreams = config
.getStoreNames
.filter(config.getChangelogStream(_).isDefined)
.map(name => (name, config.getChangelogStream(name).get)).toMap
.mapValues(Util.getSystemStreamFromNames(_))
```
And each of the store stream will use the same partition, which is fetched from this mapping.
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
<https://reviews.apache.org/r/32147/#comment132595>
boolean default in java is false. Or is there something else you are referring to ?
- Naveen Somasundaram
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Yan Fang <ya...@gmail.com>.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java, lines 60-68
> > <https://reviews.apache.org/r/32147/diff/7/?file=922539#file922539line60>
> >
> > actually, we do not need the consumer after boostrapping, right? Do we stop it?
>
> Naveen Somasundaram wrote:
> We use the same consumer for both the checkpoint and changelog. The checkpoint consumer needs to live for the entire lifespan of the Job (Hence the coordinator consumer) will live for the entire lifespan of the job (Same logic as commented why we keep bootstrapping each time).
All right, got a little confused now.
* Q1: Do we use the same coordinatorStream for both checkpoint and changelog?
* Q2: Does this consumer happen in Container or AM?
* Q3: When do we need the consumer? Only when starting the container or some other time as well?
- Yan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java, line 98
> > <https://reviews.apache.org/r/32147/diff/7/?file=922532#file922532line98>
> >
> > why do we need to bootstrap everytime we call this method?
We need to bootstrap everytime because, if the container fails and comes back up, then we need to fetch the latest checkpoint from the coordinator stream before we serve. This logic happens in JobCoordinator, for every HTTP request, we refresh the JobModel, which ensures we always have the latest JobModel (which has the latest checkpoint, because we bootstrap everytime).
- Naveen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
- Naveen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Yan Fang <ya...@gmail.com>.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, line 149
> > <https://reviews.apache.org/r/32147/diff/7/?file=922534#file922534line149>
> >
> > Will this bootstrappedStreamSet overflow or take a lot of memory ? Because we are putting all the messages in the stream to this map. When we consider the checkpoint msg, it can be huge.
>
> Naveen Somasundaram wrote:
> It is true that we'll have multiple checkpoint messages, the reason why it won't be big:
> 1. The Coordinator stream will be log compacted.
> 2. The HashSet reads from the oldest to the latest, and if there are duplicates, it will overwritten in the HashSet. i.e., the Set should only have the latest checkpoint (one per Task), all the previous checkpoints will be overwritten as you are reading newer ones.
>
> Yan Fang wrote:
> "2. The HashSet reads from the oldest to the latest, and if there are duplicates, it will overwritten in the HashSet. i.e., the Set should only have the latest checkpoint (one per Task), all the previous checkpoints will be overwritten as you are reading newer ones."
>
> Why is this? From the code,
> ```
> CoordinatorStreamMessage coordinatorStreamMessage = new CoordinatorStreamMessage(keyArray, valueMap);
> bootstrappedStreamSet.add(coordinatorStreamMessage);
> ```
> It keeps adding new coordinatorStreamMessage
>
> Naveen Somasundaram wrote:
> Yea, it's a hashset though, and when the SystemConsumer will read the message (from the oldest offset - if you look at the register method), it will give a message and the new message with the same key will overwrite the old one in the HashSet. But, I just looked at the equals method of the HashSet (CoordinatorMessage's equals method):
> ```
> @Override
> public boolean equals(Object obj) {
> if (this == obj)
> return true;
> if (obj == null)
> return false;
> if (getClass() != obj.getClass())
> return false;
> CoordinatorStreamMessage other = (CoordinatorStreamMessage) obj;
> if (isDelete != other.isDelete)
> return false;
> if (!Arrays.equals(keyArray, other.keyArray))
> return false;
> if (messageMap == null) {
> if (other.messageMap != null)
> return false;
> } else if (!messageMap.equals(other.messageMap))
> return false;
> return true;
> }
>
> ```
>
> It also uses the MessageMap for equals operations, so the checkpoint will not be compacted in the code (barring the fact that it will be log compacted in Kafka). I will have to override the equals method for CheckpointMessage and ChangelogMessage. I'll make this change with the doc. Good catch Yan!
Sounds good. Thanks for the explaination.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java, lines 60-68
> > <https://reviews.apache.org/r/32147/diff/7/?file=922539#file922539line60>
> >
> > actually, we do not need the consumer after boostrapping, right? Do we stop it?
>
> Naveen Somasundaram wrote:
> We use the same consumer for both the checkpoint and changelog. The checkpoint consumer needs to live for the entire lifespan of the Job (Hence the coordinator consumer) will live for the entire lifespan of the job (Same logic as commented why we keep bootstrapping each time).
>
> Yan Fang wrote:
> All right, got a little confused now.
> * Q1: Do we use the same coordinatorStream for both checkpoint and changelog?
> * Q2: Does this consumer happen in Container or AM?
> * Q3: When do we need the consumer? Only when starting the container or some other time as well?
>
> Naveen Somasundaram wrote:
> Your understanding is correct, it doesn't store the changelog itself, but only the mapping. The change log is still a separate stream.
> Q1: That's correct, coordinatorstream is a single stream for all samza "meta" information. (Checkpoint/ChangelogMapping/Configs) all of them are present in the same stream and the coordinatorMessage type is used to distinguish them.
> Q2: The consumption happens in the AM, and the AM generates the JobModel, which is served to the containers when they start. However, the container writes(never reads) to the coordinatorstream (only the checkpoint messages)
> Q3: We need a consumer to read all checkpoints/changelogmapping and configs. Since we need to keep reading checkpoints all the time in the AM (To server the latest to the container), pretty much the entire lifecycle of the job.
>
> I think a lot of it becomes clearer when I finish writing the documentation with diagrams and stuff.
"Q2: The consumption happens in the AM, and the AM generates the JobModel, which is served to the containers when they start. However, the container writes(never reads) to the coordinatorstream (only the checkpoint messages)"
Yeah, this is what I thought. However, if you check the changes in SamzaContainer.class, we are starting the Consumer, which is not necessary.
```
val coordinatorSystemConsumer = new CoordinatorStreamSystemFactory().getCoordinatorStreamSystemConsumer(config, samzaContainerMetrics.registry)
-->
val checkpointManager = new CheckpointManager(coordinatorSystemProducer, coordinatorSystemConsumer, String.valueOf(containerId))
-->
val offsetManager = OffsetManager(inputStreamMetadata, config, checkpointManager, systemAdmins, offsetManagerMetrics, combinedOffsets)
-->
offsetManager.start
-->
initializeCheckpointManager
-->
coordinatorStreamProducer.start();
coordinatorStreamConsumer.start();
```
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala, lines 83-92
> > <https://reviews.apache.org/r/32147/diff/7/?file=922552#file922552line83>
> >
> > Will the old config be overwritten when we produce the new config?
>
> Naveen Somasundaram wrote:
> Correct, this is code that deletes the delta (difference between the old and new config):
> ```
> (oldConfig.keySet -- config.keySet).foreach(key => {
> coordinatorSystemProducer.send(new CoordinatorStreamMessage.Delete(JobRunner.SOURCE, key, SetConfig.TYPE))
> })
> ```
>
> Yan Fang wrote:
> Here I mean, will the old config be overriden by the new config? For example we have config foo=value1 in the stream. The new producer reads config file, which has foo=value2, and then "writeConfig" to the coordinator stream. Will the foo=value1 be replaced by foo=value2, then val oldConfig = coordinatorSystemConsumer.getConfig(); will get "foo=value1" instead?
>
> Naveen Somasundaram wrote:
> oldConfig will get both the keys, but since use a map (CoordinatorStreamSystemConsumer:151-156) we will have the latest value.
> The order of events in kafka.
>
> #User add the config (foo=value1, bar=value4) and started the JobRunner
> #JobRunner writes using the producer to Kafka
> #kafka has:
> foo=value1
> bar=value4
> #User now decides to add new Configs foo=value2 and removes bar=value4
> #Starts the JobRunner new producer writes new config (line 79)
> #kafka has:
> foo=value1
> bar=value4
> foo=value2
> #now we read everything from Kafka (line 85)
> oldConfig = foo->value2, bar=value4
> newConfig = foo->value2
> #keys to be deleted
> (oldConfig -- newConfig)= bar=value4
> #kafka has:
> foo=value1
> bar=value4
> foo=value2
> bar=null (marked for compaction)
>
> The config seem by the AM:
> foo=value2
>
>
> I think oldConfig is poor name for the variable, it should really have been called consolidatedConfig. That would it make it much more readable, I'll make this change.
This makes sense. Agreed on this change. I think this is why I was misled.
- Yan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java, lines 60-68
> > <https://reviews.apache.org/r/32147/diff/7/?file=922539#file922539line60>
> >
> > actually, we do not need the consumer after boostrapping, right? Do we stop it?
>
> Naveen Somasundaram wrote:
> We use the same consumer for both the checkpoint and changelog. The checkpoint consumer needs to live for the entire lifespan of the Job (Hence the coordinator consumer) will live for the entire lifespan of the job (Same logic as commented why we keep bootstrapping each time).
>
> Yan Fang wrote:
> All right, got a little confused now.
> * Q1: Do we use the same coordinatorStream for both checkpoint and changelog?
> * Q2: Does this consumer happen in Container or AM?
> * Q3: When do we need the consumer? Only when starting the container or some other time as well?
Your understanding is correct, it doesn't store the changelog itself, but only the mapping. The change log is still a separate stream.
Q1: That's correct, coordinatorstream is a single stream for all samza "meta" information. (Checkpoint/ChangelogMapping/Configs) all of them are present in the same stream and the coordinatorMessage type is used to distinguish them.
Q2: The consumption happens in the AM, and the AM generates the JobModel, which is served to the containers when they start. However, the container writes(never reads) to the coordinatorstream (only the checkpoint messages)
Q3: We need a consumer to read all checkpoints/changelogmapping and configs. Since we need to keep reading checkpoints all the time in the AM (To server the latest to the container), pretty much the entire lifecycle of the job.
I think a lot of it becomes clearer when I finish writing the documentation with diagrams and stuff.
- Naveen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java, line 436
> > <https://reviews.apache.org/r/32147/diff/7/?file=922533#file922533line436>
> >
> > typo, SetChangelogMapping?
I'll fix this when I update the Doc.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java, line 444
> > <https://reviews.apache.org/r/32147/diff/7/?file=922533#file922533line444>
> >
> > typo: they -> the
I will fix this when I update the Doc.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, lines 132-133
> > <https://reviews.apache.org/r/32147/diff/7/?file=922534#file922534line132>
> >
> > this doc is a little vague. Though it only pays attention to the config messges, actully it's also used by other method (getBootstrappedStream) that is called from CheckpointManager.
I agree, requires clean up, I'll address this along with the docs.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, line 149
> > <https://reviews.apache.org/r/32147/diff/7/?file=922534#file922534line149>
> >
> > Will this bootstrappedStreamSet overflow or take a lot of memory ? Because we are putting all the messages in the stream to this map. When we consider the checkpoint msg, it can be huge.
It is true that we'll have multiple checkpoint messages, the reason why it won't be big:
1. The Coordinator stream will be log compacted.
2. The HashSet reads from the oldest to the latest, and if there are duplicates, it will overwritten in the HashSet. i.e., the Set should only have the latest checkpoint (one per Task), all the previous checkpoints will be overwritten as you are reading newer ones.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, line 176
> > <https://reviews.apache.org/r/32147/diff/7/?file=922534#file922534line176>
> >
> > check if (isBoostrapped)? The same question in "public Checkpoint readLastCheckpoint(TaskName taskName) "
Same answer as the previous one, we need to bootstrap each time to guarantee correctness for failed containers.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java, lines 60-68
> > <https://reviews.apache.org/r/32147/diff/7/?file=922539#file922539line60>
> >
> > actually, we do not need the consumer after boostrapping, right? Do we stop it?
We use the same consumer for both the checkpoint and changelog. The checkpoint consumer needs to live for the entire lifespan of the Job (Hence the coordinator consumer) will live for the entire lifespan of the job (Same logic as commented why we keep bootstrapping each time).
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java, line 89
> > <https://reviews.apache.org/r/32147/diff/7/?file=922539#file922539line89>
> >
> > i think the same question, do we need to bootstrap again for the changelog?
previous answer
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java, line 104
> > <https://reviews.apache.org/r/32147/diff/7/?file=922539#file922539line104>
> >
> > same question i raised, how do we differeciate more than one store?
previous answer
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala, line 108
> > <https://reviews.apache.org/r/32147/diff/7/?file=922543#file922543line108>
> >
> > use CHECKPOINT_SEGMENT_BYTES ?
That config is removed (along with the class) because checkpoint topic is no longer a thing, it's replaced by coordinator stream.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala, line 340
> > <https://reviews.apache.org/r/32147/diff/7/?file=922547#file922547line340>
> >
> > comment for this? a little confusing.
Agreed, I'll add more clarifying comments here explained why need all offsets.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala, lines 83-92
> > <https://reviews.apache.org/r/32147/diff/7/?file=922552#file922552line83>
> >
> > Will the old config be overwritten when we produce the new config?
Correct, this is code that deletes the delta (difference between the old and new config):
```
(oldConfig.keySet -- config.keySet).foreach(key => {
coordinatorSystemProducer.send(new CoordinatorStreamMessage.Delete(JobRunner.SOURCE, key, SetConfig.TYPE))
})
```
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/util/Util.scala, line 172
> > <https://reviews.apache.org/r/32147/diff/7/?file=922557#file922557line172>
> >
> > typo: jobd -> job; format is of -> format of
will fix.
- Naveen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, line 149
> > <https://reviews.apache.org/r/32147/diff/7/?file=922534#file922534line149>
> >
> > Will this bootstrappedStreamSet overflow or take a lot of memory ? Because we are putting all the messages in the stream to this map. When we consider the checkpoint msg, it can be huge.
>
> Naveen Somasundaram wrote:
> It is true that we'll have multiple checkpoint messages, the reason why it won't be big:
> 1. The Coordinator stream will be log compacted.
> 2. The HashSet reads from the oldest to the latest, and if there are duplicates, it will overwritten in the HashSet. i.e., the Set should only have the latest checkpoint (one per Task), all the previous checkpoints will be overwritten as you are reading newer ones.
>
> Yan Fang wrote:
> "2. The HashSet reads from the oldest to the latest, and if there are duplicates, it will overwritten in the HashSet. i.e., the Set should only have the latest checkpoint (one per Task), all the previous checkpoints will be overwritten as you are reading newer ones."
>
> Why is this? From the code,
> ```
> CoordinatorStreamMessage coordinatorStreamMessage = new CoordinatorStreamMessage(keyArray, valueMap);
> bootstrappedStreamSet.add(coordinatorStreamMessage);
> ```
> It keeps adding new coordinatorStreamMessage
Yea, it's a hashset though, and when the SystemConsumer will read the message (from the oldest offset - if you look at the register method), it will give a message and the new message with the same key will overwrite the old one in the HashSet. But, I just looked at the equals method of the HashSet (CoordinatorMessage's equals method):
```
@Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (obj == null)
return false;
if (getClass() != obj.getClass())
return false;
CoordinatorStreamMessage other = (CoordinatorStreamMessage) obj;
if (isDelete != other.isDelete)
return false;
if (!Arrays.equals(keyArray, other.keyArray))
return false;
if (messageMap == null) {
if (other.messageMap != null)
return false;
} else if (!messageMap.equals(other.messageMap))
return false;
return true;
}
```
It also uses the MessageMap for equals operations, so the checkpoint will not be compacted in the code (barring the fact that it will be log compacted in Kafka). I will have to override the equals method for CheckpointMessage and ChangelogMessage. I'll make this change with the doc. Good catch Yan!
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala, lines 83-92
> > <https://reviews.apache.org/r/32147/diff/7/?file=922552#file922552line83>
> >
> > Will the old config be overwritten when we produce the new config?
>
> Naveen Somasundaram wrote:
> Correct, this is code that deletes the delta (difference between the old and new config):
> ```
> (oldConfig.keySet -- config.keySet).foreach(key => {
> coordinatorSystemProducer.send(new CoordinatorStreamMessage.Delete(JobRunner.SOURCE, key, SetConfig.TYPE))
> })
> ```
>
> Yan Fang wrote:
> Here I mean, will the old config be overriden by the new config? For example we have config foo=value1 in the stream. The new producer reads config file, which has foo=value2, and then "writeConfig" to the coordinator stream. Will the foo=value1 be replaced by foo=value2, then val oldConfig = coordinatorSystemConsumer.getConfig(); will get "foo=value1" instead?
oldConfig will get both the keys, but since use a map (CoordinatorStreamSystemConsumer:151-156) we will have the latest value.
The order of events in kafka.
#User add the config (foo=value1, bar=value4) and started the JobRunner
#JobRunner writes using the producer to Kafka
#kafka has:
foo=value1
bar=value4
#User now decides to add new Configs foo=value2 and removes bar=value4
#Starts the JobRunner new producer writes new config (line 79)
#kafka has:
foo=value1
bar=value4
foo=value2
#now we read everything from Kafka (line 85)
oldConfig = foo->value2, bar=value4
newConfig = foo->value2
#keys to be deleted
(oldConfig -- newConfig)= bar=value4
#kafka has:
foo=value1
bar=value4
foo=value2
bar=null (marked for compaction)
The config seem by the AM:
foo=value2
I think oldConfig is poor name for the variable, it should really have been called consolidatedConfig. That would it make it much more readable, I'll make this change.
- Naveen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala, line 90
> > <https://reviews.apache.org/r/32147/diff/7/?file=922543#file922543line90>
> >
> > move it to the beginning of this class?
This is an mistake from copy-paste :), should have been removed, will do.
- Naveen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Yan Fang <ya...@gmail.com>.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java, line 149
> > <https://reviews.apache.org/r/32147/diff/7/?file=922534#file922534line149>
> >
> > Will this bootstrappedStreamSet overflow or take a lot of memory ? Because we are putting all the messages in the stream to this map. When we consider the checkpoint msg, it can be huge.
>
> Naveen Somasundaram wrote:
> It is true that we'll have multiple checkpoint messages, the reason why it won't be big:
> 1. The Coordinator stream will be log compacted.
> 2. The HashSet reads from the oldest to the latest, and if there are duplicates, it will overwritten in the HashSet. i.e., the Set should only have the latest checkpoint (one per Task), all the previous checkpoints will be overwritten as you are reading newer ones.
"2. The HashSet reads from the oldest to the latest, and if there are duplicates, it will overwritten in the HashSet. i.e., the Set should only have the latest checkpoint (one per Task), all the previous checkpoints will be overwritten as you are reading newer ones."
Why is this? From the code,
```
CoordinatorStreamMessage coordinatorStreamMessage = new CoordinatorStreamMessage(keyArray, valueMap);
bootstrappedStreamSet.add(coordinatorStreamMessage);
```
It keeps adding new coordinatorStreamMessage
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java, line 98
> > <https://reviews.apache.org/r/32147/diff/7/?file=922532#file922532line98>
> >
> > why do we need to bootstrap everytime we call this method?
>
> Naveen Somasundaram wrote:
> We need to bootstrap everytime because, if the container fails and comes back up, then we need to fetch the latest checkpoint from the coordinator stream before we serve. This logic happens in JobCoordinator, for every HTTP request, we refresh the JobModel, which ensures we always have the latest JobModel (which has the latest checkpoint, because we bootstrap everytime).
this makes sense. Thanks.
> On April 29, 2015, 7:30 p.m., Yan Fang wrote:
> > samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala, lines 83-92
> > <https://reviews.apache.org/r/32147/diff/7/?file=922552#file922552line83>
> >
> > Will the old config be overwritten when we produce the new config?
>
> Naveen Somasundaram wrote:
> Correct, this is code that deletes the delta (difference between the old and new config):
> ```
> (oldConfig.keySet -- config.keySet).foreach(key => {
> coordinatorSystemProducer.send(new CoordinatorStreamMessage.Delete(JobRunner.SOURCE, key, SetConfig.TYPE))
> })
> ```
Here I mean, will the old config be overriden by the new config? For example we have config foo=value1 in the stream. The new producer reads config file, which has foo=value2, and then "writeConfig" to the coordinator stream. Will the foo=value1 be replaced by foo=value2, then val oldConfig = coordinatorSystemConsumer.getConfig(); will get "foo=value1" instead?
- Yan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Yan Fang <ya...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review81995
-----------------------------------------------------------
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java
<https://reviews.apache.org/r/32147/#comment132563>
why do we need to bootstrap everytime we call this method?
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java
<https://reviews.apache.org/r/32147/#comment132566>
typo, SetChangelogMapping?
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java
<https://reviews.apache.org/r/32147/#comment132567>
typo: they -> the
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java
<https://reviews.apache.org/r/32147/#comment132564>
if this is for the changelog of the store, how do we differentiate two stores in the same TaskName and Container?
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
<https://reviews.apache.org/r/32147/#comment132568>
no default value?
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
<https://reviews.apache.org/r/32147/#comment132561>
this doc is a little vague. Though it only pays attention to the config messges, actully it's also used by other method (getBootstrappedStream) that is called from CheckpointManager.
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
<https://reviews.apache.org/r/32147/#comment132558>
Will this bootstrappedStreamSet overflow or take a lot of memory ? Because we are putting all the messages in the stream to this map. When we consider the checkpoint msg, it can be huge.
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java
<https://reviews.apache.org/r/32147/#comment132562>
check if (isBoostrapped)? The same question in "public Checkpoint readLastCheckpoint(TaskName taskName) "
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java
<https://reviews.apache.org/r/32147/#comment132569>
no default value?
samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java
<https://reviews.apache.org/r/32147/#comment132571>
actually, we do not need the consumer after boostrapping, right? Do we stop it?
samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java
<https://reviews.apache.org/r/32147/#comment132570>
i think the same question, do we need to bootstrap again for the changelog?
samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java
<https://reviews.apache.org/r/32147/#comment132572>
same question i raised, how do we differeciate more than one store?
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala
<https://reviews.apache.org/r/32147/#comment132573>
move it to the beginning of this class?
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala
<https://reviews.apache.org/r/32147/#comment132574>
use CHECKPOINT_SEGMENT_BYTES ?
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
<https://reviews.apache.org/r/32147/#comment132575>
comment for this? a little confusing.
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala
<https://reviews.apache.org/r/32147/#comment132576>
Will the old config be overwritten when we produce the new config?
samza-core/src/main/scala/org/apache/samza/util/Util.scala
<https://reviews.apache.org/r/32147/#comment132577>
typo: jobd -> job; format is of -> format of
- Yan Fang
On April 10, 2015, 3:13 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 10, 2015, 3:13 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/
-----------------------------------------------------------
(Updated April 10, 2015, 3:13 a.m.)
Review request for samza.
Changes
-------
Fixed the issues Navina discovered on Hello Samza. I ran the integration test and unit testa again. I'll throw a new RB tomorrow with all changes combined after I test it will Hello Samza.
Repository: samza
Description
-------
SAMZA 465
Diffs (updated)
-----
build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 5416dd620b2f65ffb09cf5f8c07b1a547df82bab
samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala cab31ca6452e45c73808f20b12d39a30c117119a
samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 3dfa26a23c88f9b5b8ffbfd64b59c4061a8ed2e7
samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala c84ceb75b84eb6d2ce115396ba54cf9e455d905c
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
samza-test/src/main/config/negate-number.properties b9f898c745250252461c833adb05e24ece2d4a89
samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
samza-test/src/main/config/perf/kafka-read-write-performance.properties 122b14aaecc2d221ac8944d04d508e7f83ede5ab
samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
Diff: https://reviews.apache.org/r/32147/diff/
Testing
-------
The patch does the following:
1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
3. Passes the checkpoint information to the containers from the jobcoordinator
4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
Tests:
All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
Pending Issues:
1. The metrics registry is right now not used correctly, I need to pass the right reference to it
2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
Thanks,
Naveen Somasundaram
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/
-----------------------------------------------------------
(Updated April 3, 2015, 12:52 a.m.)
Review request for samza.
Changes
-------
Addressing review comments.
Repository: samza
Description
-------
SAMZA 465
Diffs (updated)
-----
build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 87fcf589b10fd952df630d498c645159ed381739
samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 81742bc93bcbca5a2ed43c701bcd5d3f40d79bfe
samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 7fc6d89b5d703a7c10a212aaa8d3f9231996b897
samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4f15002325bc0154991f9a35312e903d15ef81e7
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
Diff: https://reviews.apache.org/r/32147/diff/
Testing
-------
The patch does the following:
1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
3. Passes the checkpoint information to the containers from the jobcoordinator
4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
Tests:
All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
Pending Issues:
1. The metrics registry is right now not used correctly, I need to pass the right reference to it
2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
Thanks,
Naveen Somasundaram
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
> On April 2, 2015, 11:22 p.m., Chris Riccomini wrote:
> > samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java, line 43
> > <https://reviews.apache.org/r/32147/diff/3-5/?file=911560#file911560line43>
> >
> > Can we make this private final, and call new in the constructor?
yes, changed.
> On April 2, 2015, 11:22 p.m., Chris Riccomini wrote:
> > samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java, line 44
> > <https://reviews.apache.org/r/32147/diff/3-5/?file=911560#file911560line44>
> >
> > All this stuff should be private final, right?
changed.
> On April 2, 2015, 11:22 p.m., Chris Riccomini wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java, line 395
> > <https://reviews.apache.org/r/32147/diff/3-5/?file=911561#file911561line395>
> >
> > TaskName or container id?
My bad, fixed.
> On April 2, 2015, 11:22 p.m., Chris Riccomini wrote:
> > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java, line 441
> > <https://reviews.apache.org/r/32147/diff/3-5/?file=911561#file911561line441>
> >
> > TaskName or container id?
fixed
> On April 2, 2015, 11:22 p.m., Chris Riccomini wrote:
> > samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java, line 39
> > <https://reviews.apache.org/r/32147/diff/3-5/?file=911567#file911567line39>
> >
> > Same comments as CheckpointManager. private finals and second constructor.
done
> On April 2, 2015, 11:22 p.m., Chris Riccomini wrote:
> > samza-core/src/main/scala/org/apache/samza/util/Util.scala, line 128
> > <https://reviews.apache.org/r/32147/diff/3-5/?file=911585#file911585line128>
> >
> > Is it just 200, or all 2XX codes that we're OK with?
I looked at the other HTTP code, http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#2xx_Success
None of the other seem consumable.
> On April 2, 2015, 11:22 p.m., Chris Riccomini wrote:
> > samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala, line 124
> > <https://reviews.apache.org/r/32147/diff/3-5/?file=911568#file911568line124>
> >
> > nit: lowercase c
changed.
> On April 2, 2015, 11:22 p.m., Chris Riccomini wrote:
> > samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala, line 75
> > <https://reviews.apache.org/r/32147/diff/3-5/?file=911576#file911576line75>
> >
> > Nit: lowercase 'j'
changed.
> On April 2, 2015, 11:22 p.m., Chris Riccomini wrote:
> > samza-core/src/main/scala/org/apache/samza/util/Util.scala, line 146
> > <https://reviews.apache.org/r/32147/diff/3-5/?file=911585#file911585line146>
> >
> > I don't think that you need to parameterize this (the {}). SLF4J's Logger takes an error(String, Throwable) .. http://www.slf4j.org/apidocs/org/slf4j/Logger.html
Thanks! was not aware of that. Changed.
- Naveen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review78680
-----------------------------------------------------------
On April 3, 2015, 12:52 a.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 3, 2015, 12:52 a.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 87fcf589b10fd952df630d498c645159ed381739
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 81742bc93bcbca5a2ed43c701bcd5d3f40d79bfe
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 7fc6d89b5d703a7c10a212aaa8d3f9231996b897
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4f15002325bc0154991f9a35312e903d15ef81e7
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Chris Riccomini <cr...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/#review78680
-----------------------------------------------------------
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java
<https://reviews.apache.org/r/32147/#comment127632>
Can we make this private final, and call new in the constructor?
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java
<https://reviews.apache.org/r/32147/#comment127631>
All this stuff should be private final, right?
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java
<https://reviews.apache.org/r/32147/#comment127630>
Prefer two constructors. One with source, and the other without.
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java
<https://reviews.apache.org/r/32147/#comment127633>
TaskName or container id?
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java
<https://reviews.apache.org/r/32147/#comment127634>
TaskName or container id?
samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java
<https://reviews.apache.org/r/32147/#comment127635>
Same comments as CheckpointManager. private finals and second constructor.
samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala
<https://reviews.apache.org/r/32147/#comment127636>
nit: lowercase c
samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala
<https://reviews.apache.org/r/32147/#comment127732>
Nit: lowercase 'j'
samza-core/src/main/scala/org/apache/samza/util/Util.scala
<https://reviews.apache.org/r/32147/#comment127734>
Is it just 200, or all 2XX codes that we're OK with?
samza-core/src/main/scala/org/apache/samza/util/Util.scala
<https://reviews.apache.org/r/32147/#comment127733>
I don't think that you need to parameterize this (the {}). SLF4J's Logger takes an error(String, Throwable) .. http://www.slf4j.org/apidocs/org/slf4j/Logger.html
- Chris Riccomini
On April 1, 2015, 9:02 p.m., Naveen Somasundaram wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32147/
> -----------------------------------------------------------
>
> (Updated April 1, 2015, 9:02 p.m.)
>
>
> Review request for samza.
>
>
> Repository: samza
>
>
> Description
> -------
>
> SAMZA 465
>
>
> Diffs
> -----
>
> build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
> samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
> samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
> samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
> samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
> samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
> samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
> samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
> samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
> samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
> samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
> samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 87fcf589b10fd952df630d498c645159ed381739
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
> samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
> samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
> samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
> samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
> samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
> samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
> samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
> samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 81742bc93bcbca5a2ed43c701bcd5d3f40d79bfe
> samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
> samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
> samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 7fc6d89b5d703a7c10a212aaa8d3f9231996b897
> samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
> samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4f15002325bc0154991f9a35312e903d15ef81e7
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
> samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
> samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
> samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
> samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
> samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
> samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
> samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
>
> Diff: https://reviews.apache.org/r/32147/diff/
>
>
> Testing
> -------
>
> The patch does the following:
> 1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
> 2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
> 3. Passes the checkpoint information to the containers from the jobcoordinator
> 4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
>
>
> Tests:
> All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
> Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
>
> Pending Issues:
> 1. The metrics registry is right now not used correctly, I need to pass the right reference to it
> 2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
> 3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
>
>
> Thanks,
>
> Naveen Somasundaram
>
>
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/
-----------------------------------------------------------
(Updated April 1, 2015, 9:02 p.m.)
Review request for samza.
Changes
-------
Rebased with master, addressed previous review comments, added retry loop for reconnecting with job coordinator, added metrics for coordinator.
Repository: samza
Description
-------
SAMZA 465
Diffs (updated)
-----
build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 87fcf589b10fd952df630d498c645159ed381739
samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 81742bc93bcbca5a2ed43c701bcd5d3f40d79bfe
samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala c9504ecf1d1edbc116ca6d794678062fddeee7fa
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 7fc6d89b5d703a7c10a212aaa8d3f9231996b897
samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala f783c5790f442928ea83b13359ac6b2a5bcb02e5
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4f15002325bc0154991f9a35312e903d15ef81e7
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 0380d35121f4feb99efc7d092b4232030d12db01
samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java d3f25c0e03a727e64a774581384ef5aae9ef9c1c
samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala d66b3bd070a4cef4b1d3dded1d79a33cbe3fa09b
samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
Diff: https://reviews.apache.org/r/32147/diff/
Testing
-------
The patch does the following:
1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
3. Passes the checkpoint information to the containers from the jobcoordinator
4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
Tests:
All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
Pending Issues:
1. The metrics registry is right now not used correctly, I need to pass the right reference to it
2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
Thanks,
Naveen Somasundaram
Re: Review Request 32147: SAMZA-465
Posted by Naveen Somasundaram <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32147/
-----------------------------------------------------------
(Updated April 1, 2015, 7:44 p.m.)
Review request for samza.
Repository: samza
Description
-------
SAMZA 465
Diffs (updated)
-----
build.gradle 0a268ac7a3819cf46b54a93e0e3171455371456a
samza-api/src/main/java/org/apache/samza/checkpoint/Checkpoint.java
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 092cb910b40d312217e86420bf1ddfbaf605e9e5
samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java a97ff0919d8205928efee1a2a20780659180849d
samza-api/src/main/java/org/apache/samza/container/TaskName.java 083358686fc69ab45bbc73e898f419224ebc3a9f
samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java 8995ba30c823bddcdfd3af7100e1440e71ef7998
samza-api/src/main/java/org/apache/samza/task/TaskCoordinator.java 6ff1a555f3c48e416bb78e94c5df71eff0a27f3d
samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java 01997ae22641b735cd452a0e89a49219e2874892
samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java eb22d2ec5f09ca59790e2871d9bff9745fe925dc
samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java 7dc431c74a3fc2ba80eb47d6c5d87524cb4c9bde
samza-core/src/main/java/org/apache/samza/serializers/model/SamzaObjectMapper.java 3517912eaafbf95f8c8cc70ab5869548a56b76e7
samza-core/src/main/java/org/apache/samza/storage/ChangelogPartitionManager.java PRE-CREATION
samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala ddc30af7c52d8a4d5c5de02f6757c040b1f31c93
samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala a40c87fa7865746a5612c55a4cf24c8d005be7e0
samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala 2a87a6e0cef72179b5383fc824266de1f9606293
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
samza-core/src/main/scala/org/apache/samza/config/ShellCommandConfig.scala 1a2dd4413f56e53dbeeb47b5637d7b0c50522f02
samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala adef09e15c666cb2dbb2e4c5507fc2d605b82a1e
samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 1ca9e2cc5673c537b6a48224809847e94da81fca
samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 275eb1a924d09a0a43efe6273e0d2af9217e1c74
samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala c14f2f623bb4bae911dd3085ce428175930e4545
samza-core/src/main/scala/org/apache/samza/coordinator/server/HttpServer.scala 10986a49b39cda703a0e54688dc914f2465c79c9
samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala 635c3531a897a369c813821f7b901186e1281ed1
samza-core/src/main/scala/org/apache/samza/coordinator/stream/CoordinatorStreamSystemFactory.scala PRE-CREATION
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala b80d34953a54ada461ed1d4b0dcfa07f4435f877
samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 530255e5866bc49ec5ce1a0b7437470cd4e17010
samza-core/src/main/scala/org/apache/samza/serializers/JsonSerde.scala 744eec05857a4ea14c718e3750fb575d3678b1f8
samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala ec1d749cb5186f788b402877996a4caa37e99362
samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586eeec95dabfeb3b6881af9b3865c3029ca
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java PRE-CREATION
samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java 76bc6810a3162a1dc58a36033b3b1f75616bd6ca
samza-core/src/test/resources/test.properties 9348c7de956ebf02f58a163dc6fb391a7e29ae64
samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala af800dfeedbfea75abaac3f15fd53bc55b743daf
samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala d18d4c45c5de3b50a24d6c776364e1f589db8f4d
samza-core/src/test/scala/org/apache/samza/checkpoint/file/TestFileSystemCheckpointManager.scala 10ff1f437220b38810f8a32903cc72df20f206d3
samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 81742bc93bcbca5a2ed43c701bcd5d3f40d79bfe
samza-core/src/test/scala/org/apache/samza/container/grouper/task/TestGroupByContainerCount.scala 1ee5c06fbd1be5e4ce944a16454c8bd32459d395
samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala a8e5d36921464a2e36693279e8083e4544c4e289
samza-core/src/test/scala/org/apache/samza/coordinator/server/TestHttpServer.scala 9a8e1450a6cc14713817f719cfa56a0e5c97a6f6
samza-core/src/test/scala/org/apache/samza/serializers/TestJsonSerde.scala 60460713a2d4f7b7b389f21c1450d45c1afaa0f4
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointLogKey.scala 5d8ee4fa74be9ba36956c11ae33573be2d2d5826
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala 4a1b31f025ba7b05a7b46041aa8e12074599ce24
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala 7fc6d89b5d703a7c10a212aaa8d3f9231996b897
samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 02a6ea94daf9eb597c9ecef5d63062df5861efc8
samza-kafka/src/main/scala/org/apache/samza/config/RegExTopicGenerator.scala a34c3f2738855dbf3737639c33846fcad23bd3b9
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala b790be17cfe08da28220ffb381cbd618ebe25cf0
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemFactory.scala 4f15002325bc0154991f9a35312e903d15ef81e7
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointLogKey.scala b76d5ad68640908bef552125d405b467386025f8
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala 7d4bea8398794c2325f9c022074303a83cfb164a
samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemAdmin.scala 3d1e6ecbb3fd95816c722a68c4f5907120eb20d0
samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java 4ef3551f470e77e27bd156e81ce96486f25c21bf
samza-test/src/main/config/join/common.properties ad10aac090beb072ecce3546f06279a7a6113970
samza-test/src/main/config/perf/container-performance.properties 86b2d58e2d4a9d6bcc9bd305bb7d735f7d1dd053
samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java c0a20af5a2f4329ad4a2cff378ced3bececbc1cb
samza-test/src/main/python/samza_failure_testing.py 198db26528cab8b473f794a922848a60299dc825
samza-test/src/main/scala/org/apache/samza/test/performance/TestPerformanceTask.scala 45c76e86361b9e1a54cb5fc717126d36b64cf7e8
samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala a8b724bf781003142e455fdf1fed2f13d6c18353
samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 03395e2efa0fec723e354177d06bfacf7d2a9215
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala a1dbe0435ae08c710d4bfc871458ed386e275cd2
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala 8ba435ef2ccf2af64d01eb4bc3b1c362fb03779d
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala b0b6543856cb87888c5a719182ad9576b51bba1a
samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 24b11da06a69da734c85720ef39d65ee46d821d5
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala 765f72f4c10bd0f1d1adab28c8ec54d9cbea5fb4
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 81dea9d6d1921462b200c62dbdf016c0eb2f01b2
samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 58f2464211a1fb7ff40f5978fd41f64d088002d0
Diff: https://reviews.apache.org/r/32147/diff/
Testing
-------
The patch does the following:
1. Removes all checkpointmanagers specific to the system and replaces them to a single entity that uses coordinator stream to do the checkpointing
2. Decoupled change log from checkpoint manager, a new component called changelog manager will do that.
3. Passes the checkpoint information to the containers from the jobcoordinator
4. Modifies the offsetmanger to use the new checkpoint manager and starting offsets from jobcoordinator.
Tests:
All existing Unit tests and Zopkio tests pass, I have changed some of existing Unit tests as well.
Plan to add one more Unit test to verify checkpoint persistence (Stub present in TestJobCoordinator)
Pending Issues:
1. The metrics registry is right now not used correctly, I need to pass the right reference to it
2. The coordinator stream does not use the same consumer and producer from the "systemconsumers" and "systemproducers" we have (This will be done after SAMZA-567)
3. Remove checkpoint configs related to the stream, migrate them to the coordinator stream and document them.
Thanks,
Naveen Somasundaram