You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Benno Evers <be...@mesosphere.com> on 2017/12/01 18:32:47 UTC

Re: Review Request 64012: Added new --reconfiguration_compatibility slave flag and implementation.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64012/
-----------------------------------------------------------

(Updated Dec. 1, 2017, 6:32 p.m.)


Review request for mesos and Vinod Kone.


Repository: mesos


Description
-------

This flag allows operators to weaken the checks performed by the agent
when recovering state, in particular it allows to recover running tasks
even when parts of the recovered SlaveInfo don't match the current
state.

The set of possible changes is quite restricted for now,
to avoid accidenetally introducing semantic bugs.


Diffs (updated)
-----

  src/CMakeLists.txt 15cda101099ce819b72d09bece37ce106b900bb0 
  src/Makefile.am 3444388e1799a6de774c41c65d273084706ed6ca 
  src/slave/compatibility.hpp PRE-CREATION 
  src/slave/compatibility.cpp PRE-CREATION 
  src/slave/flags.hpp f25d8aff29d0698eaf06dd40a86a35e43bc9e094 
  src/slave/flags.cpp 34edf4cb4a1afc366e6b6b4ddf78e90e49853fdf 
  src/slave/slave.hpp 643d8559244f0842e82b293c1ef99cb26e111059 
  src/slave/slave.cpp 3896e432fe4b40a458e75537ccfcdd689cdf3f43 
  src/tests/CMakeLists.txt 5e5202079c9798b91028e845490b747d4d1d0a66 
  src/tests/master_tests.cpp 57eae320a7a398527cd3623c89bf67f319a8e955 
  src/tests/slave_compatibility_tests.cpp PRE-CREATION 
  src/tests/slave_recovery_tests.cpp f14c6ef69eb20a03454c8197df79b572a3c6d050 
  src/tests/slave_tests.cpp 8ab63ac70083f1acaf1d253f9818efd11add6e94 


Diff: https://reviews.apache.org/r/64012/diff/5/

Changes: https://reviews.apache.org/r/64012/diff/4-5/


Testing
-------


Thanks,

Benno Evers


Re: Review Request 64012: Added new --reconfiguration_compatibility slave flag and implementation.

Posted by Benno Evers <be...@mesosphere.com>.

> On Dec. 4, 2017, 11:14 p.m., Vinod Kone wrote:
> > src/tests/slave_tests.cpp
> > Lines 8912 (patched)
> > <https://reviews.apache.org/r/64012/diff/4-6/?file=1903901#file1903901line8920>
> >
> >     what about testing the resources?

I originally left that out because it has to assume quite a lot about the internal resource handling of master and slave, which didn't seem very relevant to agent reconfiguration and I thought it would be highly unlikely that a bug would be able to fail this and pass the test in `master_tests.cpp` where we actually check the resources. Of course, it turned out that there actually was a latent bug in the allocator which happens only in precisely the situation we have here: https://reviews.apache.org/r/64342/


- Benno


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64012/#review192788
-----------------------------------------------------------


On Dec. 4, 2017, 4:33 p.m., Benno Evers wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64012/
> -----------------------------------------------------------
> 
> (Updated Dec. 4, 2017, 4:33 p.m.)
> 
> 
> Review request for mesos and Vinod Kone.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> This flag allows operators to weaken the checks performed by the agent
> when recovering state, in particular it allows to recover running tasks
> even when parts of the recovered SlaveInfo don't match the current
> state.
> 
> The set of possible changes is quite restricted for now,
> to avoid accidenetally introducing semantic bugs.
> 
> 
> Diffs
> -----
> 
>   src/CMakeLists.txt 592489df070a0c4fdee814a4a7b613e62a544f88 
>   src/Makefile.am 08d29ab58fe5c7f5e0c9173b504787ec254ed99b 
>   src/slave/compatibility.hpp PRE-CREATION 
>   src/slave/compatibility.cpp PRE-CREATION 
>   src/slave/flags.hpp f25d8aff29d0698eaf06dd40a86a35e43bc9e094 
>   src/slave/flags.cpp 34edf4cb4a1afc366e6b6b4ddf78e90e49853fdf 
>   src/slave/slave.hpp 643d8559244f0842e82b293c1ef99cb26e111059 
>   src/slave/slave.cpp c07e25f668bef6b6fa3cae4b69ba90de3eb3bdcc 
>   src/tests/CMakeLists.txt 5e5202079c9798b91028e845490b747d4d1d0a66 
>   src/tests/master_tests.cpp 08742ec878b29ef275fb40e857efffcf4c2f9ac2 
>   src/tests/slave_compatibility_tests.cpp PRE-CREATION 
>   src/tests/slave_recovery_tests.cpp 7674e607ee5bb7efbceb81c3e7291bc558d69935 
>   src/tests/slave_tests.cpp a7f6658fa467914cd74691b5631d26d85ba7c1eb 
> 
> 
> Diff: https://reviews.apache.org/r/64012/diff/6/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Benno Evers
> 
>


Re: Review Request 64012: Added new --reconfiguration_compatibility slave flag and implementation.

Posted by Vinod Kone <vi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64012/#review192788
-----------------------------------------------------------


Fix it, then Ship it!





src/tests/master_tests.cpp
Line 2738 (original), 2735 (patched)
<https://reviews.apache.org/r/64012/#comment271045>

    mvoe this to #2760



src/tests/slave_tests.cpp
Lines 8912 (patched)
<https://reviews.apache.org/r/64012/#comment271051>

    what about testing the resources?



src/tests/slave_recovery_tests.cpp
Lines 4641 (patched)
<https://reviews.apache.org/r/64012/#comment271047>

    s/tests/test/



src/tests/slave_recovery_tests.cpp
Line 4908 (original), 5043 (patched)
<https://reviews.apache.org/r/64012/#comment271048>

    single blank line.



src/tests/slave_recovery_tests.cpp
Lines 5088-5089 (patched)
<https://reviews.apache.org/r/64012/#comment271050>

    2 extra blank lines?


- Vinod Kone


On Dec. 4, 2017, 4:33 p.m., Benno Evers wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64012/
> -----------------------------------------------------------
> 
> (Updated Dec. 4, 2017, 4:33 p.m.)
> 
> 
> Review request for mesos and Vinod Kone.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> This flag allows operators to weaken the checks performed by the agent
> when recovering state, in particular it allows to recover running tasks
> even when parts of the recovered SlaveInfo don't match the current
> state.
> 
> The set of possible changes is quite restricted for now,
> to avoid accidenetally introducing semantic bugs.
> 
> 
> Diffs
> -----
> 
>   src/CMakeLists.txt 592489df070a0c4fdee814a4a7b613e62a544f88 
>   src/Makefile.am 08d29ab58fe5c7f5e0c9173b504787ec254ed99b 
>   src/slave/compatibility.hpp PRE-CREATION 
>   src/slave/compatibility.cpp PRE-CREATION 
>   src/slave/flags.hpp f25d8aff29d0698eaf06dd40a86a35e43bc9e094 
>   src/slave/flags.cpp 34edf4cb4a1afc366e6b6b4ddf78e90e49853fdf 
>   src/slave/slave.hpp 643d8559244f0842e82b293c1ef99cb26e111059 
>   src/slave/slave.cpp c07e25f668bef6b6fa3cae4b69ba90de3eb3bdcc 
>   src/tests/CMakeLists.txt 5e5202079c9798b91028e845490b747d4d1d0a66 
>   src/tests/master_tests.cpp 08742ec878b29ef275fb40e857efffcf4c2f9ac2 
>   src/tests/slave_compatibility_tests.cpp PRE-CREATION 
>   src/tests/slave_recovery_tests.cpp 7674e607ee5bb7efbceb81c3e7291bc558d69935 
>   src/tests/slave_tests.cpp a7f6658fa467914cd74691b5631d26d85ba7c1eb 
> 
> 
> Diff: https://reviews.apache.org/r/64012/diff/6/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Benno Evers
> 
>


Re: Review Request 64012: Added new `--reconfiguration_policy` slave flag and implementation.

Posted by Vinod Kone <vi...@apache.org>.

> On Dec. 5, 2017, 8:18 p.m., Vinod Kone wrote:
> > src/slave/compatibility.hpp
> > Lines 33-36 (original), 32-41 (patched)
> > <https://reviews.apache.org/r/64012/diff/6-7/?file=1908031#file1908031line33>
> >
> >     Instead of specifying each field here, since it can go out of date, why not just say every field in `SlaveInfo` because that's what you are essentially doing?

I'll fix this while committing.


- Vinod


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64012/#review192913
-----------------------------------------------------------


On Dec. 5, 2017, 4:02 p.m., Benno Evers wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64012/
> -----------------------------------------------------------
> 
> (Updated Dec. 5, 2017, 4:02 p.m.)
> 
> 
> Review request for mesos and Vinod Kone.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> This flag allows operators to select a set of permitted
> configuration changes that the slave will tolerate during
> recovery while still recovering running tasks and keeping
> the same agent id.
> 
> The previous behaviour of Mesos is reproduced exactly by setting
> this flag to "equal". For now only one additional policy is
> provided, "additive".
> 
> 
> Diffs
> -----
> 
>   src/CMakeLists.txt 592489df070a0c4fdee814a4a7b613e62a544f88 
>   src/Makefile.am 08d29ab58fe5c7f5e0c9173b504787ec254ed99b 
>   src/slave/compatibility.hpp PRE-CREATION 
>   src/slave/compatibility.cpp PRE-CREATION 
>   src/slave/flags.hpp f25d8aff29d0698eaf06dd40a86a35e43bc9e094 
>   src/slave/flags.cpp 34edf4cb4a1afc366e6b6b4ddf78e90e49853fdf 
>   src/slave/slave.hpp 643d8559244f0842e82b293c1ef99cb26e111059 
>   src/slave/slave.cpp c07e25f668bef6b6fa3cae4b69ba90de3eb3bdcc 
>   src/tests/CMakeLists.txt 5e5202079c9798b91028e845490b747d4d1d0a66 
>   src/tests/master_tests.cpp 08742ec878b29ef275fb40e857efffcf4c2f9ac2 
>   src/tests/slave_compatibility_tests.cpp PRE-CREATION 
>   src/tests/slave_recovery_tests.cpp 7674e607ee5bb7efbceb81c3e7291bc558d69935 
>   src/tests/slave_tests.cpp a7f6658fa467914cd74691b5631d26d85ba7c1eb 
> 
> 
> Diff: https://reviews.apache.org/r/64012/diff/7/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Benno Evers
> 
>


Re: Review Request 64012: Added new `--reconfiguration_policy` slave flag and implementation.

Posted by Vinod Kone <vi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64012/#review192913
-----------------------------------------------------------


Fix it, then Ship it!





src/slave/compatibility.hpp
Lines 33-36 (original), 32-41 (patched)
<https://reviews.apache.org/r/64012/#comment271257>

    Instead of specifying each field here, since it can go out of date, why not just say every field in `SlaveInfo` because that's what you are essentially doing?


- Vinod Kone


On Dec. 5, 2017, 4:02 p.m., Benno Evers wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64012/
> -----------------------------------------------------------
> 
> (Updated Dec. 5, 2017, 4:02 p.m.)
> 
> 
> Review request for mesos and Vinod Kone.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> This flag allows operators to select a set of permitted
> configuration changes that the slave will tolerate during
> recovery while still recovering running tasks and keeping
> the same agent id.
> 
> The previous behaviour of Mesos is reproduced exactly by setting
> this flag to "equal". For now only one additional policy is
> provided, "additive".
> 
> 
> Diffs
> -----
> 
>   src/CMakeLists.txt 592489df070a0c4fdee814a4a7b613e62a544f88 
>   src/Makefile.am 08d29ab58fe5c7f5e0c9173b504787ec254ed99b 
>   src/slave/compatibility.hpp PRE-CREATION 
>   src/slave/compatibility.cpp PRE-CREATION 
>   src/slave/flags.hpp f25d8aff29d0698eaf06dd40a86a35e43bc9e094 
>   src/slave/flags.cpp 34edf4cb4a1afc366e6b6b4ddf78e90e49853fdf 
>   src/slave/slave.hpp 643d8559244f0842e82b293c1ef99cb26e111059 
>   src/slave/slave.cpp c07e25f668bef6b6fa3cae4b69ba90de3eb3bdcc 
>   src/tests/CMakeLists.txt 5e5202079c9798b91028e845490b747d4d1d0a66 
>   src/tests/master_tests.cpp 08742ec878b29ef275fb40e857efffcf4c2f9ac2 
>   src/tests/slave_compatibility_tests.cpp PRE-CREATION 
>   src/tests/slave_recovery_tests.cpp 7674e607ee5bb7efbceb81c3e7291bc558d69935 
>   src/tests/slave_tests.cpp a7f6658fa467914cd74691b5631d26d85ba7c1eb 
> 
> 
> Diff: https://reviews.apache.org/r/64012/diff/7/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Benno Evers
> 
>


Re: Review Request 64012: Added new `--reconfiguration_policy` slave flag and implementation.

Posted by Benno Evers <be...@mesosphere.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64012/
-----------------------------------------------------------

(Updated Dec. 5, 2017, 4:02 p.m.)


Review request for mesos and Vinod Kone.


Summary (updated)
-----------------

Added new `--reconfiguration_policy` slave flag and implementation.


Repository: mesos


Description (updated)
-------

This flag allows operators to select a set of permitted
configuration changes that the slave will tolerate during
recovery while still recovering running tasks and keeping
the same agent id.

The previous behaviour of Mesos is reproduced exactly by setting
this flag to "equal". For now only one additional policy is
provided, "additive".


Diffs (updated)
-----

  src/CMakeLists.txt 592489df070a0c4fdee814a4a7b613e62a544f88 
  src/Makefile.am 08d29ab58fe5c7f5e0c9173b504787ec254ed99b 
  src/slave/compatibility.hpp PRE-CREATION 
  src/slave/compatibility.cpp PRE-CREATION 
  src/slave/flags.hpp f25d8aff29d0698eaf06dd40a86a35e43bc9e094 
  src/slave/flags.cpp 34edf4cb4a1afc366e6b6b4ddf78e90e49853fdf 
  src/slave/slave.hpp 643d8559244f0842e82b293c1ef99cb26e111059 
  src/slave/slave.cpp c07e25f668bef6b6fa3cae4b69ba90de3eb3bdcc 
  src/tests/CMakeLists.txt 5e5202079c9798b91028e845490b747d4d1d0a66 
  src/tests/master_tests.cpp 08742ec878b29ef275fb40e857efffcf4c2f9ac2 
  src/tests/slave_compatibility_tests.cpp PRE-CREATION 
  src/tests/slave_recovery_tests.cpp 7674e607ee5bb7efbceb81c3e7291bc558d69935 
  src/tests/slave_tests.cpp a7f6658fa467914cd74691b5631d26d85ba7c1eb 


Diff: https://reviews.apache.org/r/64012/diff/7/

Changes: https://reviews.apache.org/r/64012/diff/6-7/


Testing
-------


Thanks,

Benno Evers


Re: Review Request 64012: Added new --reconfiguration_compatibility slave flag and implementation.

Posted by Benno Evers <be...@mesosphere.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64012/
-----------------------------------------------------------

(Updated Dec. 4, 2017, 4:33 p.m.)


Review request for mesos and Vinod Kone.


Repository: mesos


Description
-------

This flag allows operators to weaken the checks performed by the agent
when recovering state, in particular it allows to recover running tasks
even when parts of the recovered SlaveInfo don't match the current
state.

The set of possible changes is quite restricted for now,
to avoid accidenetally introducing semantic bugs.


Diffs (updated)
-----

  src/CMakeLists.txt 592489df070a0c4fdee814a4a7b613e62a544f88 
  src/Makefile.am 08d29ab58fe5c7f5e0c9173b504787ec254ed99b 
  src/slave/compatibility.hpp PRE-CREATION 
  src/slave/compatibility.cpp PRE-CREATION 
  src/slave/flags.hpp f25d8aff29d0698eaf06dd40a86a35e43bc9e094 
  src/slave/flags.cpp 34edf4cb4a1afc366e6b6b4ddf78e90e49853fdf 
  src/slave/slave.hpp 643d8559244f0842e82b293c1ef99cb26e111059 
  src/slave/slave.cpp c07e25f668bef6b6fa3cae4b69ba90de3eb3bdcc 
  src/tests/CMakeLists.txt 5e5202079c9798b91028e845490b747d4d1d0a66 
  src/tests/master_tests.cpp 08742ec878b29ef275fb40e857efffcf4c2f9ac2 
  src/tests/slave_compatibility_tests.cpp PRE-CREATION 
  src/tests/slave_recovery_tests.cpp 7674e607ee5bb7efbceb81c3e7291bc558d69935 
  src/tests/slave_tests.cpp a7f6658fa467914cd74691b5631d26d85ba7c1eb 


Diff: https://reviews.apache.org/r/64012/diff/6/

Changes: https://reviews.apache.org/r/64012/diff/5-6/


Testing
-------


Thanks,

Benno Evers


Re: Review Request 64012: Added new --reconfiguration_compatibility slave flag and implementation.

Posted by Benno Evers <be...@mesosphere.com>.

> On Dec. 1, 2017, 7:47 p.m., Vinod Kone wrote:
> > src/tests/slave_recovery_tests.cpp
> > Lines 4774 (patched)
> > <https://reviews.apache.org/r/64012/diff/5/?file=1905954#file1905954line4774>
> >
> >     Can you add a test that checks that any attribute changes will result in filtered offers being resent?

Added in https://reviews.apache.org/r/64011/


- Benno


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64012/#review192539
-----------------------------------------------------------


On Dec. 1, 2017, 6:32 p.m., Benno Evers wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64012/
> -----------------------------------------------------------
> 
> (Updated Dec. 1, 2017, 6:32 p.m.)
> 
> 
> Review request for mesos and Vinod Kone.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> This flag allows operators to weaken the checks performed by the agent
> when recovering state, in particular it allows to recover running tasks
> even when parts of the recovered SlaveInfo don't match the current
> state.
> 
> The set of possible changes is quite restricted for now,
> to avoid accidenetally introducing semantic bugs.
> 
> 
> Diffs
> -----
> 
>   src/CMakeLists.txt 15cda101099ce819b72d09bece37ce106b900bb0 
>   src/Makefile.am 3444388e1799a6de774c41c65d273084706ed6ca 
>   src/slave/compatibility.hpp PRE-CREATION 
>   src/slave/compatibility.cpp PRE-CREATION 
>   src/slave/flags.hpp f25d8aff29d0698eaf06dd40a86a35e43bc9e094 
>   src/slave/flags.cpp 34edf4cb4a1afc366e6b6b4ddf78e90e49853fdf 
>   src/slave/slave.hpp 643d8559244f0842e82b293c1ef99cb26e111059 
>   src/slave/slave.cpp 3896e432fe4b40a458e75537ccfcdd689cdf3f43 
>   src/tests/CMakeLists.txt 5e5202079c9798b91028e845490b747d4d1d0a66 
>   src/tests/master_tests.cpp 57eae320a7a398527cd3623c89bf67f319a8e955 
>   src/tests/slave_compatibility_tests.cpp PRE-CREATION 
>   src/tests/slave_recovery_tests.cpp f14c6ef69eb20a03454c8197df79b572a3c6d050 
>   src/tests/slave_tests.cpp 8ab63ac70083f1acaf1d253f9818efd11add6e94 
> 
> 
> Diff: https://reviews.apache.org/r/64012/diff/5/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Benno Evers
> 
>


Re: Review Request 64012: Added new --reconfiguration_compatibility slave flag and implementation.

Posted by Vinod Kone <vi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64012/#review192539
-----------------------------------------------------------




src/tests/master_tests.cpp
Line 2758 (original), 2785 (patched)
<https://reviews.apache.org/r/64012/#comment270750>

    re-registration



src/tests/slave_tests.cpp
Lines 8828 (patched)
<https://reviews.apache.org/r/64012/#comment270752>

    you should expand the comment on what you are testing here.



src/tests/slave_tests.cpp
Line 8817 (original), 8831 (patched)
<https://reviews.apache.org/r/64012/#comment270753>

    Can you start a scheduler in this test and make sure it gets offers with new domains and resources after the restart?



src/tests/slave_recovery_tests.cpp
Lines 4774 (patched)
<https://reviews.apache.org/r/64012/#comment270751>

    Can you add a test that checks that any attribute changes will result in filtered offers being resent?


- Vinod Kone


On Dec. 1, 2017, 6:32 p.m., Benno Evers wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64012/
> -----------------------------------------------------------
> 
> (Updated Dec. 1, 2017, 6:32 p.m.)
> 
> 
> Review request for mesos and Vinod Kone.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> This flag allows operators to weaken the checks performed by the agent
> when recovering state, in particular it allows to recover running tasks
> even when parts of the recovered SlaveInfo don't match the current
> state.
> 
> The set of possible changes is quite restricted for now,
> to avoid accidenetally introducing semantic bugs.
> 
> 
> Diffs
> -----
> 
>   src/CMakeLists.txt 15cda101099ce819b72d09bece37ce106b900bb0 
>   src/Makefile.am 3444388e1799a6de774c41c65d273084706ed6ca 
>   src/slave/compatibility.hpp PRE-CREATION 
>   src/slave/compatibility.cpp PRE-CREATION 
>   src/slave/flags.hpp f25d8aff29d0698eaf06dd40a86a35e43bc9e094 
>   src/slave/flags.cpp 34edf4cb4a1afc366e6b6b4ddf78e90e49853fdf 
>   src/slave/slave.hpp 643d8559244f0842e82b293c1ef99cb26e111059 
>   src/slave/slave.cpp 3896e432fe4b40a458e75537ccfcdd689cdf3f43 
>   src/tests/CMakeLists.txt 5e5202079c9798b91028e845490b747d4d1d0a66 
>   src/tests/master_tests.cpp 57eae320a7a398527cd3623c89bf67f319a8e955 
>   src/tests/slave_compatibility_tests.cpp PRE-CREATION 
>   src/tests/slave_recovery_tests.cpp f14c6ef69eb20a03454c8197df79b572a3c6d050 
>   src/tests/slave_tests.cpp 8ab63ac70083f1acaf1d253f9818efd11add6e94 
> 
> 
> Diff: https://reviews.apache.org/r/64012/diff/5/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Benno Evers
> 
>