You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Naveen Somasundaram <na...@gmail.com> on 2015/06/20 02:05:29 UTC

Review Request 35676: Checkpoint migration

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35676/
-----------------------------------------------------------

Review request for samza.


Repository: samza


Description
-------

SAMZA-615: Checkpoint migration


Diffs
-----

  samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java 6c1e488d00d8593d59c89b57e673e0b6b90fd7d2 
  samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 
  samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/migration/MigrationPlan.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java 647cadb3a4e51bec8204197d77ad35a6b29afcec 
  samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java e454593ebd2b09f0cfb7a7531f05394348253e9b 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointLogKey.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManager.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManagerFactory.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointMigration.scala PRE-CREATION 
  samza-kafka/src/test/scala/old/checkpoint/TestKafkaCheckpointManager.scala PRE-CREATION 
  samza-kafka/src/test/scala/old/checkpoint/TestKafkaCheckpointMigration.scala PRE-CREATION 
  samza-test/src/main/config/join/common.properties ac87e81aeea4015b2fc83942a62e3d16c4fbbd2b 

Diff: https://reviews.apache.org/r/35676/diff/


Testing
-------

Unit test added, test will hello-samza in progress


Thanks,

Naveen Somasundaram


Re: Review Request 35676: Checkpoint migration

Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35676/#review90976
-----------------------------------------------------------


@Naveen, overall looks good. I just have one point in the previous review comment. Thanks!


samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManager.scala (line 236)
<https://reviews.apache.org/r/35676/#comment144132>

    nit: "reading both changelog partition mapping and checkpoint"?


- Yi Pan (Data Infrastructure)


On July 8, 2015, 1:41 a.m., Naveen Somasundaram wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35676/
> -----------------------------------------------------------
> 
> (Updated July 8, 2015, 1:41 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> SAMZA-615: Checkpoint migration
> 
> 
> Diffs
> -----
> 
>   build.gradle a5f54106a822dc91ff82270df27217a8765a0d80 
>   checkstyle/import-control.xml 3374f0c432e61ac4cda275377604cfd481f0cddf 
>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java 6c1e488d00d8593d59c89b57e673e0b6b90fd7d2 
>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java 92f8907f3cbd490ccafb2d963091a644604eb49b 
>   samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 
>   samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/migration/MigrationPlan.java PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java 647cadb3a4e51bec8204197d77ad35a6b29afcec 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java e454593ebd2b09f0cfb7a7531f05394348253e9b 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java ac26a015596688a1a9fe5a078ce506de64364938 
>   samza-core/src/test/scala/org/apache/samza/job/TestJobRunner.scala 52057edbbcd08b7d6edde7e898466d26534f52f6 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointLogKey.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManager.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManagerFactory.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointMigration.scala PRE-CREATION 
>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala 39c54aa4ee27736c0400d295904872ca6fa3fb88 
>   samza-kafka/src/test/scala/old/checkpoint/TestKafkaCheckpointManager.scala PRE-CREATION 
>   samza-test/src/main/config/join/common.properties ac87e81aeea4015b2fc83942a62e3d16c4fbbd2b 
> 
> Diff: https://reviews.apache.org/r/35676/diff/
> 
> 
> Testing
> -------
> 
> Unit test added, test will hello-samza in progress
> 
> Hello-samza test:
> 
> OUTPUT FROM 0.9:
> ~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 --from-beginning                                                                                                                                                              nsomasun@nsomasun-mn1
> 
> {"Partition 0":0}
> {}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1674","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1808","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1950","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"2103","stream":"wikipedia-raw"}}
> 
> ------------------------------------------------------------
> 
> OUTPUT FROM 0.10.0:
> 
> ~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_coordinator_wikipedia-parser_1 --from-beginning                                                                                                                                                                       nsomasun@nsomasun-mn1
> {"host":"172.21.136.130","source":"CHECKPOINTMIGRATION","values":{"kafka.wikipedia-raw.0":"2103"},"username":"nsomasun","timestamp":1435870948879}
> 
> 
> Thanks,
> 
> Naveen Somasundaram
> 
>


Re: Review Request 35676: Checkpoint migration

Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35676/#review90966
-----------------------------------------------------------


nit: could you also link this RB back to the JIRA? Thanks!

- Yi Pan (Data Infrastructure)


On July 8, 2015, 1:41 a.m., Naveen Somasundaram wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35676/
> -----------------------------------------------------------
> 
> (Updated July 8, 2015, 1:41 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> SAMZA-615: Checkpoint migration
> 
> 
> Diffs
> -----
> 
>   build.gradle a5f54106a822dc91ff82270df27217a8765a0d80 
>   checkstyle/import-control.xml 3374f0c432e61ac4cda275377604cfd481f0cddf 
>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java 6c1e488d00d8593d59c89b57e673e0b6b90fd7d2 
>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java 92f8907f3cbd490ccafb2d963091a644604eb49b 
>   samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 
>   samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/migration/MigrationPlan.java PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java 647cadb3a4e51bec8204197d77ad35a6b29afcec 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java e454593ebd2b09f0cfb7a7531f05394348253e9b 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java ac26a015596688a1a9fe5a078ce506de64364938 
>   samza-core/src/test/scala/org/apache/samza/job/TestJobRunner.scala 52057edbbcd08b7d6edde7e898466d26534f52f6 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointLogKey.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManager.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManagerFactory.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointMigration.scala PRE-CREATION 
>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala 39c54aa4ee27736c0400d295904872ca6fa3fb88 
>   samza-kafka/src/test/scala/old/checkpoint/TestKafkaCheckpointManager.scala PRE-CREATION 
>   samza-test/src/main/config/join/common.properties ac87e81aeea4015b2fc83942a62e3d16c4fbbd2b 
> 
> Diff: https://reviews.apache.org/r/35676/diff/
> 
> 
> Testing
> -------
> 
> Unit test added, test will hello-samza in progress
> 
> Hello-samza test:
> 
> OUTPUT FROM 0.9:
> ~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 --from-beginning                                                                                                                                                              nsomasun@nsomasun-mn1
> 
> {"Partition 0":0}
> {}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1674","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1808","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1950","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"2103","stream":"wikipedia-raw"}}
> 
> ------------------------------------------------------------
> 
> OUTPUT FROM 0.10.0:
> 
> ~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_coordinator_wikipedia-parser_1 --from-beginning                                                                                                                                                                       nsomasun@nsomasun-mn1
> {"host":"172.21.136.130","source":"CHECKPOINTMIGRATION","values":{"kafka.wikipedia-raw.0":"2103"},"username":"nsomasun","timestamp":1435870948879}
> 
> 
> Thanks,
> 
> Naveen Somasundaram
> 
>


Re: Review Request 35676: Checkpoint migration

Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.

> On July 8, 2015, 7:34 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala, line 46
> > <https://reviews.apache.org/r/35676/diff/3/?file=1001916#file1001916line46>
> >
> >     This reminds me of one thing: what if the job has both Kafka and Databus streams as inputs? The matching cases for Kafka will go ahead to migrate all Kafka offsets but the JobRunner will proceed w/o migrating the non-Kafka offsets. Won't it be better to abort here instead of proceeding w/ the migration? Hence, for jobs w/ mixed input streams, we will force the following safe sequence:
> >     1. run the checkpoint tool to migrate the offsets
> >     2. remove all checkpoint.factory configuration
> >     3. restart the job

Correction to my previous point: there should be no mixed checkpoint factories, hence the point on the mixed input systems is invalid. But I would think that for non-Kafka checkpoint system, it would still be safer to abort the migration, and force the safe sequence I described above.


- Yi


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35676/#review90962
-----------------------------------------------------------


On July 8, 2015, 1:41 a.m., Naveen Somasundaram wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35676/
> -----------------------------------------------------------
> 
> (Updated July 8, 2015, 1:41 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> SAMZA-615: Checkpoint migration
> 
> 
> Diffs
> -----
> 
>   build.gradle a5f54106a822dc91ff82270df27217a8765a0d80 
>   checkstyle/import-control.xml 3374f0c432e61ac4cda275377604cfd481f0cddf 
>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java 6c1e488d00d8593d59c89b57e673e0b6b90fd7d2 
>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java 92f8907f3cbd490ccafb2d963091a644604eb49b 
>   samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 
>   samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/migration/MigrationPlan.java PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java 647cadb3a4e51bec8204197d77ad35a6b29afcec 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java e454593ebd2b09f0cfb7a7531f05394348253e9b 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java ac26a015596688a1a9fe5a078ce506de64364938 
>   samza-core/src/test/scala/org/apache/samza/job/TestJobRunner.scala 52057edbbcd08b7d6edde7e898466d26534f52f6 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointLogKey.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManager.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManagerFactory.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointMigration.scala PRE-CREATION 
>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala 39c54aa4ee27736c0400d295904872ca6fa3fb88 
>   samza-kafka/src/test/scala/old/checkpoint/TestKafkaCheckpointManager.scala PRE-CREATION 
>   samza-test/src/main/config/join/common.properties ac87e81aeea4015b2fc83942a62e3d16c4fbbd2b 
> 
> Diff: https://reviews.apache.org/r/35676/diff/
> 
> 
> Testing
> -------
> 
> Unit test added, test will hello-samza in progress
> 
> Hello-samza test:
> 
> OUTPUT FROM 0.9:
> ~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 --from-beginning                                                                                                                                                              nsomasun@nsomasun-mn1
> 
> {"Partition 0":0}
> {}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1674","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1808","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1950","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"2103","stream":"wikipedia-raw"}}
> 
> ------------------------------------------------------------
> 
> OUTPUT FROM 0.10.0:
> 
> ~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_coordinator_wikipedia-parser_1 --from-beginning                                                                                                                                                                       nsomasun@nsomasun-mn1
> {"host":"172.21.136.130","source":"CHECKPOINTMIGRATION","values":{"kafka.wikipedia-raw.0":"2103"},"username":"nsomasun","timestamp":1435870948879}
> 
> 
> Thanks,
> 
> Naveen Somasundaram
> 
>


Re: Review Request 35676: Checkpoint migration

Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35676/#review90962
-----------------------------------------------------------


@Naveen, I had a question regarding to the migration w/ mixed inputs from Kafka and non-Kafka systems, which I think that should be discussed earlier. I am still continuing w/ the rest of code review.


samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java (line 319)
<https://reviews.apache.org/r/35676/#comment144118>

    nit: it would be better to add some comments here referring to SAMZA-731, and explain what's the effect of the current change.



samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java (line 528)
<https://reviews.apache.org/r/35676/#comment144119>

    Shouldn't we set the type and "migration-info" in the key array at least?



samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala (line 31)
<https://reviews.apache.org/r/35676/#comment144120>

    nit: typo here. Should be checkpointMigration().



samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala (line 46)
<https://reviews.apache.org/r/35676/#comment144125>

    This reminds me of one thing: what if the job has both Kafka and Databus streams as inputs? The matching cases for Kafka will go ahead to migrate all Kafka offsets but the JobRunner will proceed w/o migrating the non-Kafka offsets. Won't it be better to abort here instead of proceeding w/ the migration? Hence, for jobs w/ mixed input streams, we will force the following safe sequence:
    1. run the checkpoint tool to migrate the offsets
    2. remove all checkpoint.factory configuration
    3. restart the job


- Yi Pan (Data Infrastructure)


On July 8, 2015, 1:41 a.m., Naveen Somasundaram wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35676/
> -----------------------------------------------------------
> 
> (Updated July 8, 2015, 1:41 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> SAMZA-615: Checkpoint migration
> 
> 
> Diffs
> -----
> 
>   build.gradle a5f54106a822dc91ff82270df27217a8765a0d80 
>   checkstyle/import-control.xml 3374f0c432e61ac4cda275377604cfd481f0cddf 
>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java 6c1e488d00d8593d59c89b57e673e0b6b90fd7d2 
>   samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java 92f8907f3cbd490ccafb2d963091a644604eb49b 
>   samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 
>   samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/migration/MigrationPlan.java PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java 647cadb3a4e51bec8204197d77ad35a6b29afcec 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java e454593ebd2b09f0cfb7a7531f05394348253e9b 
>   samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java ac26a015596688a1a9fe5a078ce506de64364938 
>   samza-core/src/test/scala/org/apache/samza/job/TestJobRunner.scala 52057edbbcd08b7d6edde7e898466d26534f52f6 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointLogKey.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManager.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManagerFactory.scala PRE-CREATION 
>   samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointMigration.scala PRE-CREATION 
>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala 39c54aa4ee27736c0400d295904872ca6fa3fb88 
>   samza-kafka/src/test/scala/old/checkpoint/TestKafkaCheckpointManager.scala PRE-CREATION 
>   samza-test/src/main/config/join/common.properties ac87e81aeea4015b2fc83942a62e3d16c4fbbd2b 
> 
> Diff: https://reviews.apache.org/r/35676/diff/
> 
> 
> Testing
> -------
> 
> Unit test added, test will hello-samza in progress
> 
> Hello-samza test:
> 
> OUTPUT FROM 0.9:
> ~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 --from-beginning                                                                                                                                                              nsomasun@nsomasun-mn1
> 
> {"Partition 0":0}
> {}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1674","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1808","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1950","stream":"wikipedia-raw"}}
> {"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"2103","stream":"wikipedia-raw"}}
> 
> ------------------------------------------------------------
> 
> OUTPUT FROM 0.10.0:
> 
> ~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_coordinator_wikipedia-parser_1 --from-beginning                                                                                                                                                                       nsomasun@nsomasun-mn1
> {"host":"172.21.136.130","source":"CHECKPOINTMIGRATION","values":{"kafka.wikipedia-raw.0":"2103"},"username":"nsomasun","timestamp":1435870948879}
> 
> 
> Thanks,
> 
> Naveen Somasundaram
> 
>


Re: Review Request 35676: Checkpoint migration

Posted by Naveen Somasundaram <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35676/
-----------------------------------------------------------

(Updated July 8, 2015, 1:41 a.m.)


Review request for samza.


Repository: samza


Description
-------

SAMZA-615: Checkpoint migration


Diffs (updated)
-----

  build.gradle a5f54106a822dc91ff82270df27217a8765a0d80 
  checkstyle/import-control.xml 3374f0c432e61ac4cda275377604cfd481f0cddf 
  samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java 6c1e488d00d8593d59c89b57e673e0b6b90fd7d2 
  samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemProducer.java 92f8907f3cbd490ccafb2d963091a644604eb49b 
  samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 
  samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/migration/MigrationPlan.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java 647cadb3a4e51bec8204197d77ad35a6b29afcec 
  samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java e454593ebd2b09f0cfb7a7531f05394348253e9b 
  samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamMessage.java ac26a015596688a1a9fe5a078ce506de64364938 
  samza-core/src/test/scala/org/apache/samza/job/TestJobRunner.scala 52057edbbcd08b7d6edde7e898466d26534f52f6 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointLogKey.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManager.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManagerFactory.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointMigration.scala PRE-CREATION 
  samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala 39c54aa4ee27736c0400d295904872ca6fa3fb88 
  samza-kafka/src/test/scala/old/checkpoint/TestKafkaCheckpointManager.scala PRE-CREATION 
  samza-test/src/main/config/join/common.properties ac87e81aeea4015b2fc83942a62e3d16c4fbbd2b 

Diff: https://reviews.apache.org/r/35676/diff/


Testing
-------

Unit test added, test will hello-samza in progress

Hello-samza test:

OUTPUT FROM 0.9:
~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 --from-beginning                                                                                                                                                              nsomasun@nsomasun-mn1

{"Partition 0":0}
{}
{"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1674","stream":"wikipedia-raw"}}
{"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1808","stream":"wikipedia-raw"}}
{"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1950","stream":"wikipedia-raw"}}
{"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"2103","stream":"wikipedia-raw"}}

------------------------------------------------------------

OUTPUT FROM 0.10.0:

~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_coordinator_wikipedia-parser_1 --from-beginning                                                                                                                                                                       nsomasun@nsomasun-mn1
{"host":"172.21.136.130","source":"CHECKPOINTMIGRATION","values":{"kafka.wikipedia-raw.0":"2103"},"username":"nsomasun","timestamp":1435870948879}


Thanks,

Naveen Somasundaram


Re: Review Request 35676: Checkpoint migration

Posted by Naveen Somasundaram <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35676/
-----------------------------------------------------------

(Updated July 2, 2015, 9:10 p.m.)


Review request for samza.


Repository: samza


Description
-------

SAMZA-615: Checkpoint migration


Diffs
-----

  build.gradle a5f54106a822dc91ff82270df27217a8765a0d80 
  checkstyle/import-control.xml 3374f0c432e61ac4cda275377604cfd481f0cddf 
  samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java 6c1e488d00d8593d59c89b57e673e0b6b90fd7d2 
  samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 
  samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/migration/MigrationPlan.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java 647cadb3a4e51bec8204197d77ad35a6b29afcec 
  samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java e454593ebd2b09f0cfb7a7531f05394348253e9b 
  samza-core/src/test/scala/org/apache/samza/job/TestJobRunner.scala 52057edbbcd08b7d6edde7e898466d26534f52f6 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointLogKey.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManager.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManagerFactory.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointMigration.scala PRE-CREATION 
  samza-kafka/src/test/scala/old/checkpoint/TestKafkaCheckpointManager.scala PRE-CREATION 
  samza-test/src/main/config/join/common.properties ac87e81aeea4015b2fc83942a62e3d16c4fbbd2b 

Diff: https://reviews.apache.org/r/35676/diff/


Testing (updated)
-------

Unit test added, test will hello-samza in progress

Hello-samza test:

OUTPUT FROM 0.9:
~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 --from-beginning                                                                                                                                                              nsomasun@nsomasun-mn1

{"Partition 0":0}
{}
{"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1674","stream":"wikipedia-raw"}}
{"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1808","stream":"wikipedia-raw"}}
{"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"1950","stream":"wikipedia-raw"}}
{"SystemStreamPartition [kafka, wikipedia-raw, 0]":{"system":"kafka","partition":"0","offset":"2103","stream":"wikipedia-raw"}}

------------------------------------------------------------

OUTPUT FROM 0.10.0:

~/Documents/hello-samza(branch:latest) » deploy/kafka/bin/kafka-console-consumer.sh  --zookeeper localhost:2181 --topic __samza_coordinator_wikipedia-parser_1 --from-beginning                                                                                                                                                                       nsomasun@nsomasun-mn1
{"host":"172.21.136.130","source":"CHECKPOINTMIGRATION","values":{"kafka.wikipedia-raw.0":"2103"},"username":"nsomasun","timestamp":1435870948879}


Thanks,

Naveen Somasundaram


Re: Review Request 35676: Checkpoint migration

Posted by Naveen Somasundaram <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35676/
-----------------------------------------------------------

(Updated June 24, 2015, 6:46 p.m.)


Review request for samza.


Repository: samza


Description
-------

SAMZA-615: Checkpoint migration


Diffs (updated)
-----

  build.gradle a5f54106a822dc91ff82270df27217a8765a0d80 
  checkstyle/import-control.xml 3374f0c432e61ac4cda275377604cfd481f0cddf 
  samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java 6c1e488d00d8593d59c89b57e673e0b6b90fd7d2 
  samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 
  samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/migration/MigrationPlan.java PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java 647cadb3a4e51bec8204197d77ad35a6b29afcec 
  samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java e454593ebd2b09f0cfb7a7531f05394348253e9b 
  samza-core/src/test/scala/org/apache/samza/job/TestJobRunner.scala 52057edbbcd08b7d6edde7e898466d26534f52f6 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointLogKey.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManager.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointManagerFactory.scala PRE-CREATION 
  samza-kafka/src/main/scala/old/checkpoint/KafkaCheckpointMigration.scala PRE-CREATION 
  samza-kafka/src/test/scala/old/checkpoint/TestKafkaCheckpointManager.scala PRE-CREATION 
  samza-test/src/main/config/join/common.properties ac87e81aeea4015b2fc83942a62e3d16c4fbbd2b 

Diff: https://reviews.apache.org/r/35676/diff/


Testing
-------

Unit test added, test will hello-samza in progress


Thanks,

Naveen Somasundaram