You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Kapil Arya <ka...@mesosphere.io> on 2018/04/20 20:19:47 UTC

Review Request 66746: Replaced protobuf-specific comparators with MessageDifferencer.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66746/
-----------------------------------------------------------

Review request for mesos, Benjamin Mahler and Till Toenshoff.


Repository: mesos


Description
-------

The MessageDifferencer utility has been available since protobuf 3.0.0.
When comparing most repeated fields, we treat them as sets so the order
of elements doesn't matter. In some exceptional cases, we treat them as
list so the order becomes important. Further, fields with default
purposes are considered to be set for comparison purposes.


Diffs
-----

  include/mesos/type_utils.hpp 19ea81716496bcc0117a1b0ff157a0374f38bbfa 
  include/mesos/v1/mesos.hpp fda3eb42061f820869a2d8da939fccadc4e5ddfb 
  src/common/type_utils.cpp 33d63809b61a18e03ff745c88f024c71dd221ca2 
  src/tests/master_allocator_tests.cpp e1aef8a9625a805e7ad2dfad37bfeedee82f160d 
  src/v1/mesos.cpp 9b2df2dd798dff24a91a2604ab53c7fbb5ecfbcf 


Diff: https://reviews.apache.org/r/66746/diff/1/


Testing
-------

`make check` currently fails with:
    DefaultExecutorCheckTest.CommandCheckSharesWorkDirWithTask
    DefaultExecutorCheckTest.MultipleTasksWithChecks


Thanks,

Kapil Arya


Re: Review Request 66746: Replaced protobuf-specific comparators with MessageDifferencer.

Posted by Benjamin Mahler <bm...@apache.org>.

> On April 24, 2018, 12:43 a.m., Benjamin Mahler wrote:
> > I'm not sure this approach is tenable, we may in the future have two different `foo` fields with differing equality semantics.
> > 
> > I was expecting that this patch would add an equality operator for specific types we want to check. Do we need the `Message&` level equality operator?
> 
> Kapil Arya wrote:
>     Right now we don't have any differing equality semantics for any of the messages. There are four non-standard semantics are captured in the generic method. In future, when we need differring techniques for some bar, we can overload the `==` operator for that particular type. Does that sound reasonable?
>     
>     Since, MessageDifferencer recurses down to the fields without calling `==` itself, if we were to have two message Bar and Baz with with different equality semantics for `foo` fields in them , we'd can implement `==` for both Bar and Baz to reflect that. For example:
>     ```
>     operator==(const Bar& left, const Bar& right) {
>       MessageDifferencer diff;
>       
>       // Bar-specific enhancements for Foo.
>       diff.TreatAsSet(Foo::descriptor()->FindFieldByName("some_field"));
>       ...
>       return diff.compare(left, right);
>     }
>     
>     operator==(const Baz& left, const Baz& right) {
>       MessageDifferencer diff;
>       
>       // Baz-speficic enhancements for Foo.
>       diff.TreatAsList(Foo::descriptor()->FindFieldByName("some_field"));
>       ...
>       return diff.compare(left, right);
>     }
>     ```
>     
>     However, if Bar and Baz were to have another common field `qux` with a non-standard comparator, we'd have to duplicate the qux-specific enhancement in both Bar and Baz comparators. This leads to some duplication, but that can be fixed with some clever maneuvering.
> 
> Kapil Arya wrote:
>     There is another possibility where we implement some generic MesosMessageDifferencer like the one in the patch. Any implementations that need to deviate from the standard then take a copy of MesosMessageDifferencer and modify it as needed.

The generic + overload approach seems prone to mistakes:

```
operator ==(Message& m1, Message& m2) { /* default message differencer */ }
operator ==(Bar& b1, Bar& b2) { /* custom message differencer */ };
bool some_generic_func(Message& m1, Message& m2) { return m1 == m2 }

Bar b1, b2;

if (some_generic_func(b1, b2)) {
  // used generic comparison, whoops!
}

if (b1 == b2) {
  // used custom comparison
}
```

We would have to be very careful forbidding generic equality being used where custom equality exists and that seems rather hard to enforce? It would be even worse if the generic one encoded something custom for some fields that happened to have name collisions with Bar's fields.

Also, I think if we were to have a top level generic equality on `Message&`, it should only use the default differencer behavior.

> There is another possibility where we implement some generic MesosMessageDifferencer like the one in the patch.

Can you clarify what problem that solves? It just seems like that will be a giant monstrosity of many fields from all the protos. I assume you're trying to avoid the field specific logic duplication? Aren't there other ways to solve that problem (e.g. functions that can be called that mutate the differencer? each == operator only calls into the ones it cares about?).

I think an easier starting point here might be to modify the ones you need to start with and we can address the broader change in a non-blocking manner with the rest of your chain here?


- Benjamin


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66746/#review201799
-----------------------------------------------------------


On April 20, 2018, 8:19 p.m., Kapil Arya wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66746/
> -----------------------------------------------------------
> 
> (Updated April 20, 2018, 8:19 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Till Toenshoff.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> The MessageDifferencer utility has been available since protobuf 3.0.0.
> When comparing most repeated fields, we treat them as sets so the order
> of elements doesn't matter. In some exceptional cases, we treat them as
> list so the order becomes important. Further, fields with default
> purposes are considered to be set for comparison purposes.
> 
> 
> Diffs
> -----
> 
>   include/mesos/type_utils.hpp 19ea81716496bcc0117a1b0ff157a0374f38bbfa 
>   include/mesos/v1/mesos.hpp fda3eb42061f820869a2d8da939fccadc4e5ddfb 
>   src/common/type_utils.cpp 33d63809b61a18e03ff745c88f024c71dd221ca2 
>   src/tests/master_allocator_tests.cpp e1aef8a9625a805e7ad2dfad37bfeedee82f160d 
>   src/v1/mesos.cpp 9b2df2dd798dff24a91a2604ab53c7fbb5ecfbcf 
> 
> 
> Diff: https://reviews.apache.org/r/66746/diff/1/
> 
> 
> Testing
> -------
> 
> `make check` currently fails with:
>     DefaultExecutorCheckTest.CommandCheckSharesWorkDirWithTask
>     DefaultExecutorCheckTest.MultipleTasksWithChecks
> 
> 
> Thanks,
> 
> Kapil Arya
> 
>


Re: Review Request 66746: Replaced protobuf-specific comparators with MessageDifferencer.

Posted by Kapil Arya <ka...@mesosphere.io>.

> On April 23, 2018, 8:43 p.m., Benjamin Mahler wrote:
> > I'm not sure this approach is tenable, we may in the future have two different `foo` fields with differing equality semantics.
> > 
> > I was expecting that this patch would add an equality operator for specific types we want to check. Do we need the `Message&` level equality operator?

Right now we don't have any differing equality semantics for any of the messages. There are four non-standard semantics are captured in the generic method. In future, when we need differring techniques for some bar, we can overload the `==` operator for that particular type. Does that sound reasonable?

Since, MessageDifferencer recurses down to the fields without calling `==` itself, if we were to have two message Bar and Baz with with different equality semantics for `foo` fields in them , we'd can implement `==` for both Bar and Baz to reflect that. For example:
```
operator==(const Bar& left, const Bar& right) {
  MessageDifferencer diff;
  
  // Bar-specific enhancements for Foo.
  diff.TreatAsSet(Foo::descriptor()->FindFieldByName("some_field"));
  ...
  return diff.compare(left, right);
}

operator==(const Baz& left, const Baz& right) {
  MessageDifferencer diff;
  
  // Baz-speficic enhancements for Foo.
  diff.TreatAsList(Foo::descriptor()->FindFieldByName("some_field"));
  ...
  return diff.compare(left, right);
}
```

However, if Bar and Baz were to have another common field `qux` with a non-standard comparator, we'd have to duplicate the qux-specific enhancement in both Bar and Baz comparators. This leads to some duplication, but that can be fixed with some clever maneuvering.


- Kapil


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66746/#review201799
-----------------------------------------------------------


On April 20, 2018, 4:19 p.m., Kapil Arya wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66746/
> -----------------------------------------------------------
> 
> (Updated April 20, 2018, 4:19 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Till Toenshoff.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> The MessageDifferencer utility has been available since protobuf 3.0.0.
> When comparing most repeated fields, we treat them as sets so the order
> of elements doesn't matter. In some exceptional cases, we treat them as
> list so the order becomes important. Further, fields with default
> purposes are considered to be set for comparison purposes.
> 
> 
> Diffs
> -----
> 
>   include/mesos/type_utils.hpp 19ea81716496bcc0117a1b0ff157a0374f38bbfa 
>   include/mesos/v1/mesos.hpp fda3eb42061f820869a2d8da939fccadc4e5ddfb 
>   src/common/type_utils.cpp 33d63809b61a18e03ff745c88f024c71dd221ca2 
>   src/tests/master_allocator_tests.cpp e1aef8a9625a805e7ad2dfad37bfeedee82f160d 
>   src/v1/mesos.cpp 9b2df2dd798dff24a91a2604ab53c7fbb5ecfbcf 
> 
> 
> Diff: https://reviews.apache.org/r/66746/diff/1/
> 
> 
> Testing
> -------
> 
> `make check` currently fails with:
>     DefaultExecutorCheckTest.CommandCheckSharesWorkDirWithTask
>     DefaultExecutorCheckTest.MultipleTasksWithChecks
> 
> 
> Thanks,
> 
> Kapil Arya
> 
>


Re: Review Request 66746: Replaced protobuf-specific comparators with MessageDifferencer.

Posted by Kapil Arya <ka...@mesosphere.io>.

> On April 23, 2018, 8:43 p.m., Benjamin Mahler wrote:
> > I'm not sure this approach is tenable, we may in the future have two different `foo` fields with differing equality semantics.
> > 
> > I was expecting that this patch would add an equality operator for specific types we want to check. Do we need the `Message&` level equality operator?
> 
> Kapil Arya wrote:
>     Right now we don't have any differing equality semantics for any of the messages. There are four non-standard semantics are captured in the generic method. In future, when we need differring techniques for some bar, we can overload the `==` operator for that particular type. Does that sound reasonable?
>     
>     Since, MessageDifferencer recurses down to the fields without calling `==` itself, if we were to have two message Bar and Baz with with different equality semantics for `foo` fields in them , we'd can implement `==` for both Bar and Baz to reflect that. For example:
>     ```
>     operator==(const Bar& left, const Bar& right) {
>       MessageDifferencer diff;
>       
>       // Bar-specific enhancements for Foo.
>       diff.TreatAsSet(Foo::descriptor()->FindFieldByName("some_field"));
>       ...
>       return diff.compare(left, right);
>     }
>     
>     operator==(const Baz& left, const Baz& right) {
>       MessageDifferencer diff;
>       
>       // Baz-speficic enhancements for Foo.
>       diff.TreatAsList(Foo::descriptor()->FindFieldByName("some_field"));
>       ...
>       return diff.compare(left, right);
>     }
>     ```
>     
>     However, if Bar and Baz were to have another common field `qux` with a non-standard comparator, we'd have to duplicate the qux-specific enhancement in both Bar and Baz comparators. This leads to some duplication, but that can be fixed with some clever maneuvering.

There is another possibility where we implement some generic MesosMessageDifferencer like the one in the patch. Any implementations that need to deviate from the standard then take a copy of MesosMessageDifferencer and modify it as needed.


- Kapil


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66746/#review201799
-----------------------------------------------------------


On April 20, 2018, 4:19 p.m., Kapil Arya wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66746/
> -----------------------------------------------------------
> 
> (Updated April 20, 2018, 4:19 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Till Toenshoff.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> The MessageDifferencer utility has been available since protobuf 3.0.0.
> When comparing most repeated fields, we treat them as sets so the order
> of elements doesn't matter. In some exceptional cases, we treat them as
> list so the order becomes important. Further, fields with default
> purposes are considered to be set for comparison purposes.
> 
> 
> Diffs
> -----
> 
>   include/mesos/type_utils.hpp 19ea81716496bcc0117a1b0ff157a0374f38bbfa 
>   include/mesos/v1/mesos.hpp fda3eb42061f820869a2d8da939fccadc4e5ddfb 
>   src/common/type_utils.cpp 33d63809b61a18e03ff745c88f024c71dd221ca2 
>   src/tests/master_allocator_tests.cpp e1aef8a9625a805e7ad2dfad37bfeedee82f160d 
>   src/v1/mesos.cpp 9b2df2dd798dff24a91a2604ab53c7fbb5ecfbcf 
> 
> 
> Diff: https://reviews.apache.org/r/66746/diff/1/
> 
> 
> Testing
> -------
> 
> `make check` currently fails with:
>     DefaultExecutorCheckTest.CommandCheckSharesWorkDirWithTask
>     DefaultExecutorCheckTest.MultipleTasksWithChecks
> 
> 
> Thanks,
> 
> Kapil Arya
> 
>


Re: Review Request 66746: Replaced protobuf-specific comparators with MessageDifferencer.

Posted by Benjamin Mahler <bm...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66746/#review201799
-----------------------------------------------------------



I'm not sure this approach is tenable, we may in the future have two different `foo` fields with differing equality semantics.

I was expecting that this patch would add an equality operator for specific types we want to check. Do we need the `Message&` level equality operator?

- Benjamin Mahler


On April 20, 2018, 8:19 p.m., Kapil Arya wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66746/
> -----------------------------------------------------------
> 
> (Updated April 20, 2018, 8:19 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Till Toenshoff.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> The MessageDifferencer utility has been available since protobuf 3.0.0.
> When comparing most repeated fields, we treat them as sets so the order
> of elements doesn't matter. In some exceptional cases, we treat them as
> list so the order becomes important. Further, fields with default
> purposes are considered to be set for comparison purposes.
> 
> 
> Diffs
> -----
> 
>   include/mesos/type_utils.hpp 19ea81716496bcc0117a1b0ff157a0374f38bbfa 
>   include/mesos/v1/mesos.hpp fda3eb42061f820869a2d8da939fccadc4e5ddfb 
>   src/common/type_utils.cpp 33d63809b61a18e03ff745c88f024c71dd221ca2 
>   src/tests/master_allocator_tests.cpp e1aef8a9625a805e7ad2dfad37bfeedee82f160d 
>   src/v1/mesos.cpp 9b2df2dd798dff24a91a2604ab53c7fbb5ecfbcf 
> 
> 
> Diff: https://reviews.apache.org/r/66746/diff/1/
> 
> 
> Testing
> -------
> 
> `make check` currently fails with:
>     DefaultExecutorCheckTest.CommandCheckSharesWorkDirWithTask
>     DefaultExecutorCheckTest.MultipleTasksWithChecks
> 
> 
> Thanks,
> 
> Kapil Arya
> 
>


Re: Review Request 66746: Replaced protobuf-specific comparators with MessageDifferencer.

Posted by Mesos Reviewbot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66746/#review213233
-----------------------------------------------------------



Bad review!

Reviews applied: [66746]

Error:
2019-02-26 21:33:43 URL:https://reviews.apache.org/r/66746/diff/raw/ [45557/45557] -> "66746.patch" [1]
error: patch failed: include/mesos/type_utils.hpp:23
error: include/mesos/type_utils.hpp: patch does not apply
error: patch failed: include/mesos/v1/mesos.hpp:23
error: include/mesos/v1/mesos.hpp: patch does not apply
error: patch failed: src/common/type_utils.cpp:36
error: src/common/type_utils.cpp: patch does not apply
error: patch failed: src/v1/mesos.cpp:35
error: src/v1/mesos.cpp: patch does not apply

- Mesos Reviewbot


On April 20, 2018, 1:19 p.m., Kapil Arya wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66746/
> -----------------------------------------------------------
> 
> (Updated April 20, 2018, 1:19 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Till Toenshoff.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> The MessageDifferencer utility has been available since protobuf 3.0.0.
> When comparing most repeated fields, we treat them as sets so the order
> of elements doesn't matter. In some exceptional cases, we treat them as
> list so the order becomes important. Further, fields with default
> purposes are considered to be set for comparison purposes.
> 
> 
> Diffs
> -----
> 
>   include/mesos/type_utils.hpp 19ea81716496bcc0117a1b0ff157a0374f38bbfa 
>   include/mesos/v1/mesos.hpp fda3eb42061f820869a2d8da939fccadc4e5ddfb 
>   src/common/type_utils.cpp 33d63809b61a18e03ff745c88f024c71dd221ca2 
>   src/tests/master_allocator_tests.cpp e1aef8a9625a805e7ad2dfad37bfeedee82f160d 
>   src/v1/mesos.cpp 9b2df2dd798dff24a91a2604ab53c7fbb5ecfbcf 
> 
> 
> Diff: https://reviews.apache.org/r/66746/diff/1/
> 
> 
> Testing
> -------
> 
> `make check` currently fails with:
>     DefaultExecutorCheckTest.CommandCheckSharesWorkDirWithTask
>     DefaultExecutorCheckTest.MultipleTasksWithChecks
> 
> 
> Thanks,
> 
> Kapil Arya
> 
>


Re: Review Request 66746: Replaced protobuf-specific comparators with MessageDifferencer.

Posted by Mesos Reviewbot Windows <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66746/#review213669
-----------------------------------------------------------



FAIL: Failed to apply the current review.

Failed command: `python.exe .\support\apply-reviews.py -n -r 66746`

All the build artifacts available at: http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2932/mesos-review-66746

Relevant logs:

- [apply-review-66746.log](http://dcos-win.westus2.cloudapp.azure.com/artifacts/mesos-reviewbot-testing/2932/mesos-review-66746/logs/apply-review-66746.log):

```
error: patch failed: include/mesos/type_utils.hpp:23
error: include/mesos/type_utils.hpp: patch does not apply
error: patch failed: include/mesos/v1/mesos.hpp:23
error: include/mesos/v1/mesos.hpp: patch does not apply
error: patch failed: src/common/type_utils.cpp:36
error: src/common/type_utils.cpp: patch does not apply
error: patch failed: src/v1/mesos.cpp:35
error: src/v1/mesos.cpp: patch does not apply
```

- Mesos Reviewbot Windows


On April 20, 2018, 8:19 p.m., Kapil Arya wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66746/
> -----------------------------------------------------------
> 
> (Updated April 20, 2018, 8:19 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Till Toenshoff.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> The MessageDifferencer utility has been available since protobuf 3.0.0.
> When comparing most repeated fields, we treat them as sets so the order
> of elements doesn't matter. In some exceptional cases, we treat them as
> list so the order becomes important. Further, fields with default
> purposes are considered to be set for comparison purposes.
> 
> 
> Diffs
> -----
> 
>   include/mesos/type_utils.hpp 19ea81716496bcc0117a1b0ff157a0374f38bbfa 
>   include/mesos/v1/mesos.hpp fda3eb42061f820869a2d8da939fccadc4e5ddfb 
>   src/common/type_utils.cpp 33d63809b61a18e03ff745c88f024c71dd221ca2 
>   src/tests/master_allocator_tests.cpp e1aef8a9625a805e7ad2dfad37bfeedee82f160d 
>   src/v1/mesos.cpp 9b2df2dd798dff24a91a2604ab53c7fbb5ecfbcf 
> 
> 
> Diff: https://reviews.apache.org/r/66746/diff/1/
> 
> 
> Testing
> -------
> 
> `make check` currently fails with:
>     DefaultExecutorCheckTest.CommandCheckSharesWorkDirWithTask
>     DefaultExecutorCheckTest.MultipleTasksWithChecks
> 
> 
> Thanks,
> 
> Kapil Arya
> 
>