You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Xinyu Liu <xi...@gmail.com> on 2016/07/14 00:43:38 UTC
Re: Review Request 48213: SAMZA-960: Make system producer thread safe
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48213/
-----------------------------------------------------------
(Updated July 14, 2016, 12:43 a.m.)
Review request for samza, Chris Pettitt, Navina Ramesh, and Yi Pan (Data Infrastructure).
Changes
-------
Some refactoring
Repository: samza
Description
-------
All the system producers need to be thread safe in order to be used in multithreaded tasks. The following are the changes (ElasticSearchSystemProducer is already thread safe so no change made there):
In KafkaSystemProducer, remove the buggy retry logic and treat any exception as fatal.
In HdfsSystemProducer, add synchronization lock to all public methods.
Diffs (updated)
-----
samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala e9a51083aff4dc316e94144f6242fe702ca73a68
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducer.scala 1f4b5c46436e44b7c7cd1a49689c4f43f1f6ed1b
samza-kafka/src/main/scala/org/apache/samza/migration/KafkaCheckpointMigration.scala 5e8cc65260b6961350c64ddc13b9807dca9099c5
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala 3769e103616dc0f1fd869706cc086e24cd926c48
samza-kafka/src/test/java/org/apache/samza/system/kafka/TestKafkaSystemProducerJava.java 04c9113fd6c3dd56c49ff46c8c1c0ff12f68e5e2
samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemProducer.scala 8e32bba6ced090f0fc8d4e5176fe0788df36981d
Diff: https://reviews.apache.org/r/48213/diff/
Testing
-------
Unit tests and local testing.
Thanks,
Xinyu Liu
Re: Review Request 48213: SAMZA-960: Make system producer thread safe
Posted by Chris Pettitt <cp...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48213/#review142418
-----------------------------------------------------------
Ship it!
- Chris Pettitt
On July 14, 2016, 12:43 a.m., Xinyu Liu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48213/
> -----------------------------------------------------------
>
> (Updated July 14, 2016, 12:43 a.m.)
>
>
> Review request for samza, Chris Pettitt, Navina Ramesh, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> All the system producers need to be thread safe in order to be used in multithreaded tasks. The following are the changes (ElasticSearchSystemProducer is already thread safe so no change made there):
>
> In KafkaSystemProducer, remove the buggy retry logic and treat any exception as fatal.
> In HdfsSystemProducer, add synchronization lock to all public methods.
>
>
> Diffs
> -----
>
> samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala e9a51083aff4dc316e94144f6242fe702ca73a68
> samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducer.scala 1f4b5c46436e44b7c7cd1a49689c4f43f1f6ed1b
> samza-kafka/src/main/scala/org/apache/samza/migration/KafkaCheckpointMigration.scala 5e8cc65260b6961350c64ddc13b9807dca9099c5
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala 3769e103616dc0f1fd869706cc086e24cd926c48
> samza-kafka/src/test/java/org/apache/samza/system/kafka/TestKafkaSystemProducerJava.java 04c9113fd6c3dd56c49ff46c8c1c0ff12f68e5e2
> samza-kafka/src/test/scala/org/apache/samza/system/kafka/TestKafkaSystemProducer.scala 8e32bba6ced090f0fc8d4e5176fe0788df36981d
>
> Diff: https://reviews.apache.org/r/48213/diff/
>
>
> Testing
> -------
>
> Unit tests and local testing.
>
>
> Thanks,
>
> Xinyu Liu
>
>