Re: Review Request 48356: RFC: Samza as a library

[Navina Ramesh <nr...@linkedin.com>]

> On July 15, 2016, 12:12 a.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/java/org/apache/samza/processor/StreamProcessor.java, line 130
> > <https://reviews.apache.org/r/48356/diff/8/?file=1441900#file1441900line130>
> >
> >     Not sure whether we want to keep JmxServer life-cycle within the StreamProcessor life-cycle. This actually could be one thing that is shared w/ the whole JVM process and it can be passed in to the StreamProcessor.
> 
> Navina Ramesh wrote:
>     Oh I see. So, will the users create an instance of JmxServer in samza library? We could make it default, I guess and not have the user configure anything.

Yi..I tried making the JmxServer to be single  entity shared with the whole JVM process. Looks like the "JvmServer" class is defined in org.apache.samza.metrics . It serves as a wrapper around the regular jmx server. So, are you suggesting that the users pass their JmxServer instance or that we should create an instance of JmxServer within a StreamProcessor and share it between the container and JobCoordinator?


- Navina


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48356/#review142183
-----------------------------------------------------------


On July 13, 2016, 9:58 p.m., Navina Ramesh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48356/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 9:58 p.m.)
> 
> 
> Review request for samza, Chris Pettitt and Yi Pan (Data Infrastructure).
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Added ConfigBuilder and support classes
> 
> Added JobCoordinator interfaces
> 
> 
> Adding StreamProcessor, StandaloneJobCoordinator and updating SamzaContainer interface
> 
> 
> Added TestStreamProcessor and some unit tests for ConfigBuilders
> 
> 
> Changing who defined processorId
> 
> 
> Fixed checkstyle errors
> 
> 
> Replaced SamzaException with ConfigException
> 
> 
> Removing localityManager instantiation from Samza Container
> 
> 
> Diffs
> -----
> 
>   build.gradle ba4a9d14fe24e1ff170873920cd5eeef656955af 
>   checkstyle/import-control.xml 325c38131047836dc8aedaea4187598ef3ba7666 
>   samza-core/src/main/java/org/apache/samza/config/TaskConfigJava.java 021d42a70179f5d14f51ac87cb09dcc97218095e 
>   samza-core/src/main/java/org/apache/samza/configbuilder/CheckpointConfig.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/configbuilder/ConfigBuilder.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/configbuilder/GenericConfigBuilder.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/configbuilder/KafkaCheckpointConfig.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/configbuilder/KafkaSystemConfig.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/configbuilder/SerdeConfig.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/configbuilder/StandaloneConfigBuilder.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/configbuilder/SystemConfig.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/container/grouper/stream/AllSspToSingleTaskGrouper.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/container/grouper/stream/AllSspToSingleTaskGrouperFactory.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouper.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/container/grouper/task/SingleContainerGrouperFactory.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/coordinator/JobCoordinator.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/coordinator/JobCoordinatorFactory.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/processor/StreamProcessor.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/standalone/StandaloneJobCoordinator.java PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/standalone/StandaloneJobCoordinatorFactory.java PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 49b08f6b68dbb44757dcc8ce8d60c365a9d22981 
>   samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 08a4debb06f9925ae741049abb2ee0df97b2243b 
>   samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala cf05c15c836ddfa54ba8fe27abc18ed88ac5fc11 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 18c09224bbae959342daf9b2b7a7d971cc224f48 
>   samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala d3bd9b7c11afd44ccfb681b660fefffafd216c29 
>   samza-core/src/main/scala/org/apache/samza/job/local/ThreadJobFactory.scala 56881d46be9f859999adabbbda20433b208e012e 
>   samza-core/src/test/java/org/apache/samza/configbuilder/TestStandaloneConfigBuilder.java PRE-CREATION 
>   samza-test/src/test/java/org/apache/samza/processor/MyStreamTask.java PRE-CREATION 
>   samza-test/src/test/java/org/apache/samza/processor/TestStreamProcessor.java PRE-CREATION 
>   samza-yarn/src/main/java/org/apache/samza/config/YarnConfig.java 8f2dc4853a2b5dd712f25a2d2d16402bcba89d7a 
>   samza-yarn/src/main/java/org/apache/samza/job/yarn/SamzaTaskManager.java bc95f31c0dcaaa68d483a6f152b61aba6c543fff 
> 
> Diff: https://reviews.apache.org/r/48356/diff/
> 
> 
> Testing
> -------
> 
> ./gradlew clean build
> 
> Local integration test:
> ./bin/grid start zookeeper
> ./bin/grid start kafka
> Then, run TestStreamProcessor.java
> 
> 
> Thanks,
> 
> Navina Ramesh
> 
>