You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Robert Žuljevi? <r....@levi9.com> on 2015/06/04 15:50:10 UTC

Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35067/
-----------------------------------------------------------

Review request for samza.


Repository: samza


Description
-------

Added config max.in.flight.requests.per.connection and retries options for Kafka producers.


Diffs
-----

  samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala a1de88781a8ff2a3fe50389ae7a59062289404fc 

Diff: https://reviews.apache.org/r/35067/diff/


Testing
-------


Thanks,

Robert Žuljevi?


Re: Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

Posted by Navina Ramesh <nr...@linkedin.com>.

> On June 8, 2015, 2:29 a.m., Navina Ramesh wrote:
> > samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala, line 182
> > <https://reviews.apache.org/r/35067/diff/1/?file=979072#file979072line182>
> >
> >     This warning is valid only when the configured value for max.in.flight.requests.per.connection > 1. Can you check the value and make the comment more elaborate?
> 
> Robert Zuljevic wrote:
>     Hi. In order to do that I would need to cast the string to integer which can throw an NumberFormatException. Is this ok?

Yeah. I understand your concern here. I think config validation is not yet standardized in the codebase. I think this should be fixed when we address SAMZA-471


- Navina


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35067/#review86980
-----------------------------------------------------------


On June 15, 2015, 1:42 p.m., Robert Zuljevic wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35067/
> -----------------------------------------------------------
> 
> (Updated June 15, 2015, 1:42 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Fixed test
> 
> 
> Diffs
> -----
> 
>   samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala a1de88781a8ff2a3fe50389ae7a59062289404fc 
>   samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala 82006960adc45a92a79e28f078ebd5eb01d3205e 
> 
> Diff: https://reviews.apache.org/r/35067/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Zuljevic
> 
>


Re: Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

Posted by Robert Zuljevic <r....@levi9.com>.

> On June 8, 2015, 2:29 a.m., Navina Ramesh wrote:
> > samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala, line 182
> > <https://reviews.apache.org/r/35067/diff/1/?file=979072#file979072line182>
> >
> >     This warning is valid only when the configured value for max.in.flight.requests.per.connection > 1. Can you check the value and make the comment more elaborate?

Hi. In order to do that I would need to cast the string to integer which can throw an NumberFormatException. Is this ok?


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35067/#review86980
-----------------------------------------------------------


On June 4, 2015, 1:50 p.m., Robert Zuljevic wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35067/
> -----------------------------------------------------------
> 
> (Updated June 4, 2015, 1:50 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Added config max.in.flight.requests.per.connection and retries options for Kafka producers.
> 
> 
> Diffs
> -----
> 
>   samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala a1de88781a8ff2a3fe50389ae7a59062289404fc 
> 
> Diff: https://reviews.apache.org/r/35067/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Zuljevic
> 
>


Re: Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35067/#review86980
-----------------------------------------------------------


Apart from the comments, lgtm!


samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala
<https://reviews.apache.org/r/35067/#comment139190>

    This warning is valid only when the configured value for max.in.flight.requests.per.connection > 1. Can you check the value and make the comment more elaborate?



samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala
<https://reviews.apache.org/r/35067/#comment139191>

    We need a log message when this value is overriden (to < INT.MAX) and that Samza cannot handle producer failures. Hence, the onus of retries is left up to the kafka producer.


- Navina Ramesh


On June 4, 2015, 1:50 p.m., Robert Zuljevic wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35067/
> -----------------------------------------------------------
> 
> (Updated June 4, 2015, 1:50 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Added config max.in.flight.requests.per.connection and retries options for Kafka producers.
> 
> 
> Diffs
> -----
> 
>   samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala a1de88781a8ff2a3fe50389ae7a59062289404fc 
> 
> Diff: https://reviews.apache.org/r/35067/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Zuljevic
> 
>


Re: Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

Posted by Robert Zuljevic <r....@levi9.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35067/
-----------------------------------------------------------

(Updated June 16, 2015, 7:47 a.m.)


Review request for samza.


Repository: samza


Description (updated)
-------

Added tests for new Kafka config properties


Diffs (updated)
-----

  samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala a1de88781a8ff2a3fe50389ae7a59062289404fc 
  samza-kafka/src/test/scala/org/apache/samza/config/TestKafkaConfig.scala 95f76b61f70058bb567fe4eff3a3bb2298a0e197 
  samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala 82006960adc45a92a79e28f078ebd5eb01d3205e 

Diff: https://reviews.apache.org/r/35067/diff/


Testing
-------


Thanks,

Robert Zuljevic


Re: Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35067/#review87952
-----------------------------------------------------------

Ship it!


Ship It!

- Navina Ramesh


On June 15, 2015, 1:42 p.m., Robert Zuljevic wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35067/
> -----------------------------------------------------------
> 
> (Updated June 15, 2015, 1:42 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Fixed test
> 
> 
> Diffs
> -----
> 
>   samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala a1de88781a8ff2a3fe50389ae7a59062289404fc 
>   samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala 82006960adc45a92a79e28f078ebd5eb01d3205e 
> 
> Diff: https://reviews.apache.org/r/35067/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Zuljevic
> 
>


Re: Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

Posted by Navina Ramesh <nr...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35067/#review87963
-----------------------------------------------------------


@Robert:  
Can you please add a unit test in TestKafkaConfig to verify that these properties get overridden correctly ? Sorry about this afterthought request.

- Navina Ramesh


On June 15, 2015, 1:42 p.m., Robert Zuljevic wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35067/
> -----------------------------------------------------------
> 
> (Updated June 15, 2015, 1:42 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> Fixed test
> 
> 
> Diffs
> -----
> 
>   samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala a1de88781a8ff2a3fe50389ae7a59062289404fc 
>   samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala 82006960adc45a92a79e28f078ebd5eb01d3205e 
> 
> Diff: https://reviews.apache.org/r/35067/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Zuljevic
> 
>


Re: Review Request 35067: SAMZA-546: Add config option max.in.flight.requests.per.connection

Posted by Robert Zuljevic <r....@levi9.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35067/
-----------------------------------------------------------

(Updated June 15, 2015, 1:42 p.m.)


Review request for samza.


Repository: samza


Description (updated)
-------

Fixed test


Diffs (updated)
-----

  samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala a1de88781a8ff2a3fe50389ae7a59062289404fc 
  samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala 82006960adc45a92a79e28f078ebd5eb01d3205e 

Diff: https://reviews.apache.org/r/35067/diff/


Testing
-------


Thanks,

Robert Zuljevic