You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Chris Riccomini <cr...@apache.org> on 2015/02/25 01:54:07 UTC
Review Request 31392: SAMZA-555
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31392/
-----------------------------------------------------------
Review request for samza.
Bugs: SAMZA-555
https://issues.apache.org/jira/browse/SAMZA-555
Repository: samza
Description
-------
fix comments
add notes
clear assignments after coordinator is elected.
add logging
make job runner block until job finishes
add standalone job and standalone job factory. make zk work with hierarchical chroot paths.
add unsubscribe to coordinator controller
unsubscribe to assignment changes in container listener
both tests pass.
fix bug in container controller to announce ownership
add initial container controller
make test do asserts
standalone main method works as expected.
rework assignTasksToContainers. controller isn't watching /containers sequential ids yet, but it should.
fiddling around with CLI test
writing a little main method to play with zk and coordinator
initial coordinator draft
add missing leadership check
working on zk coordinator controller.
messing around
sketch out a hacky controller.
adding core dependency
create samza-standalone project
Diffs
-----
build.gradle b803276c13280ffed5ca1ce8ea608d81010e9b6b
gradle/dependency-versions.gradle 84be50b216e7dc3c430e0979a933d846f8ebbb8d
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 499f5c6a58ff88ef9105b7b6ca168215fb82f35c
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 16345cd1c1354a0d25a0000d81a307dbe3abbe81
samza-log4j/src/main/java/org/apache/samza/logging/log4j/serializers/LoggingEventStringSerde.java f9a0960735705f49f46f2e5b373ba4d33aadcb47
samza-standalone/src/main/java/org/apache/samza/job/standalone/StandaloneJob.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/job/standalone/StandaloneJobFactory.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkContainerController.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkCoordinatorController.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkCoordinatorState.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/serializers/zk/ZkJsonSerde.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/util/ZkUtil.java PRE-CREATION
samza-test/src/main/java/org/apache/samza/system/mock/MockSystemConsumer.java a282dbb2976cb916d41649c3fbe070008c6621ee
samza-test/src/main/java/org/apache/samza/task/MockTask.java PRE-CREATION
samza-test/src/test/java/org/apache/samza/job/standalone/controller/TestStandaloneZkCoordinatorController.java PRE-CREATION
settings.gradle bb07a3b84b14dcef94da1bb166eab6aa3d0026bb
Diff: https://reviews.apache.org/r/31392/diff/
Testing
-------
Thanks,
Chris Riccomini
Re: Review Request 31392: SAMZA-555
Posted by Chris Riccomini <cr...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31392/
-----------------------------------------------------------
(Updated March 8, 2015, 7:39 p.m.)
Review request for samza.
Bugs: SAMZA-555
https://issues.apache.org/jira/browse/SAMZA-555
Repository: samza
Description (updated)
-------
merge master
don't npe if a container has no assignments
killing zk works
listen to existing containers when becoming leader
spelling mistake
fix comments
add notes
clear assignments after coordinator is elected.
add logging
make job runner block until job finishes
add standalone job and standalone job factory. make zk work with hierarchical chroot paths.
add unsubscribe to coordinator controller
unsubscribe to assignment changes in container listener
both tests pass.
fix bug in container controller to announce ownership
add initial container controller
make test do asserts
standalone main method works as expected.
rework assignTasksToContainers. controller isn't watching /containers sequential ids yet, but it should.
fiddling around with CLI test
writing a little main method to play with zk and coordinator
initial coordinator draft
add missing leadership check
working on zk coordinator controller.
messing around
sketch out a hacky controller.
adding core dependency
create samza-standalone project
Diffs (updated)
-----
build.gradle 0a268ac7a3819cf46b54a93e0e3171455371456a
gradle/dependency-versions.gradle 84be50b216e7dc3c430e0979a933d846f8ebbb8d
samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 499f5c6a58ff88ef9105b7b6ca168215fb82f35c
samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 0b720ec4dd83c71fd1ce5071571c7a10babf0ddc
samza-log4j/src/main/java/org/apache/samza/logging/log4j/serializers/LoggingEventStringSerde.java 8d8f5e8a8e5fd1e4d9e5482a5accd4a7ece463bc
samza-standalone/src/main/java/org/apache/samza/job/standalone/StandaloneJob.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/job/standalone/StandaloneJobFactory.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkContainerController.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkCoordinatorController.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkCoordinatorState.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/serializers/zk/ZkJsonSerde.java PRE-CREATION
samza-standalone/src/main/java/org/apache/samza/util/ZkUtil.java PRE-CREATION
samza-test/src/main/java/org/apache/samza/system/mock/MockSystemConsumer.java a282dbb2976cb916d41649c3fbe070008c6621ee
samza-test/src/main/java/org/apache/samza/task/MockTask.java PRE-CREATION
samza-test/src/test/java/org/apache/samza/job/standalone/controller/TestStandaloneZkCoordinatorController.java PRE-CREATION
settings.gradle bb07a3b84b14dcef94da1bb166eab6aa3d0026bb
Diff: https://reviews.apache.org/r/31392/diff/
Testing
-------
Thanks,
Chris Riccomini
Re: Review Request 31392: SAMZA-555
Posted by Yan Fang <ya...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31392/#review74231
-----------------------------------------------------------
samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala
<https://reviews.apache.org/r/31392/#comment120752>
TODO: update the doc accordingly
samza-standalone/src/main/java/org/apache/samza/job/standalone/StandaloneJobFactory.java
<https://reviews.apache.org/r/31392/#comment120754>
TODO: add to configuration table
Also: move to jobConfig ?
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkContainerController.java
<https://reviews.apache.org/r/31392/#comment120762>
do we need a timeout here?
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkContainerController.java
<https://reviews.apache.org/r/31392/#comment120757>
Is it possible that the path has already existed when the job is started second time?
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkCoordinatorController.java
<https://reviews.apache.org/r/31392/#comment120759>
Do we check the path existence here? Though createPersistent does not throw exception if the path already exists, I think it's better to know the information.
samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkCoordinatorController.java
<https://reviews.apache.org/r/31392/#comment120760>
Why is it "null" here but "Collections.emptyList()" in StandaloneZkContainerController ?
samza-test/src/test/java/org/apache/samza/job/standalone/controller/TestStandaloneZkCoordinatorController.java
<https://reviews.apache.org/r/31392/#comment120763>
this is just for draft, right? :)
- Yan Fang
On Feb. 25, 2015, 12:54 a.m., Chris Riccomini wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31392/
> -----------------------------------------------------------
>
> (Updated Feb. 25, 2015, 12:54 a.m.)
>
>
> Review request for samza.
>
>
> Bugs: SAMZA-555
> https://issues.apache.org/jira/browse/SAMZA-555
>
>
> Repository: samza
>
>
> Description
> -------
>
> fix comments
>
>
> add notes
>
>
> clear assignments after coordinator is elected.
>
>
> add logging
>
>
> make job runner block until job finishes
>
>
> add standalone job and standalone job factory. make zk work with hierarchical chroot paths.
>
>
> add unsubscribe to coordinator controller
>
>
> unsubscribe to assignment changes in container listener
>
>
> both tests pass.
>
>
> fix bug in container controller to announce ownership
>
>
> add initial container controller
>
>
> make test do asserts
>
>
> standalone main method works as expected.
>
>
> rework assignTasksToContainers. controller isn't watching /containers sequential ids yet, but it should.
>
>
> fiddling around with CLI test
>
>
> writing a little main method to play with zk and coordinator
>
>
> initial coordinator draft
>
>
> add missing leadership check
>
>
> working on zk coordinator controller.
>
>
> messing around
>
>
> sketch out a hacky controller.
>
>
> adding core dependency
>
>
> create samza-standalone project
>
>
> Diffs
> -----
>
> build.gradle b803276c13280ffed5ca1ce8ea608d81010e9b6b
> gradle/dependency-versions.gradle 84be50b216e7dc3c430e0979a933d846f8ebbb8d
> samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 3b6685e00837a4aaf809813e62b7e52823bc07a9
> samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala 499f5c6a58ff88ef9105b7b6ca168215fb82f35c
> samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 16345cd1c1354a0d25a0000d81a307dbe3abbe81
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/serializers/LoggingEventStringSerde.java f9a0960735705f49f46f2e5b373ba4d33aadcb47
> samza-standalone/src/main/java/org/apache/samza/job/standalone/StandaloneJob.java PRE-CREATION
> samza-standalone/src/main/java/org/apache/samza/job/standalone/StandaloneJobFactory.java PRE-CREATION
> samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkContainerController.java PRE-CREATION
> samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkCoordinatorController.java PRE-CREATION
> samza-standalone/src/main/java/org/apache/samza/job/standalone/controller/StandaloneZkCoordinatorState.java PRE-CREATION
> samza-standalone/src/main/java/org/apache/samza/serializers/zk/ZkJsonSerde.java PRE-CREATION
> samza-standalone/src/main/java/org/apache/samza/util/ZkUtil.java PRE-CREATION
> samza-test/src/main/java/org/apache/samza/system/mock/MockSystemConsumer.java a282dbb2976cb916d41649c3fbe070008c6621ee
> samza-test/src/main/java/org/apache/samza/task/MockTask.java PRE-CREATION
> samza-test/src/test/java/org/apache/samza/job/standalone/controller/TestStandaloneZkCoordinatorController.java PRE-CREATION
> settings.gradle bb07a3b84b14dcef94da1bb166eab6aa3d0026bb
>
> Diff: https://reviews.apache.org/r/31392/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Chris Riccomini
>
>