[jira] [Comment Edited] (MINIFICPP-954) Consider moving the C2 line protocol to Google Protobuf

["Mr TheSegfault (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MINIFICPP-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893988#comment-16893988 ] 

Mr TheSegfault edited comment on MINIFICPP-954 at 7/26/19 5:04 PM:
-------------------------------------------------------------------

[~bakaid]  I like some of the concepts you present but prior CVEs were why we moved away from protobuf initially in community discussions.  I think offloading the testing to protobuf maintainers is a reason to have Protobuf but I think both will need to exist.  We also considered flatbuffers and capn proto – would love to see that added to this discussion.

 I think this ultimately would be either adding protobuf/flatbuffers/capn as a selectable line format. We have a very simple line format that isn't going to change ( this is because our line formats are designed to use the RESTFul API for the complicated stuff ). The line format is simply for sending a very tiny subset of information. We also want to use the same code for nanofi, where I think our consumers are more likely to appreciate not including protobuf. As a result we can add this as an extension ( in fact, I love the idea ) – especially since we can mitigate concerns of others by having it be configurable in the agent ( and compile time for nanofi ECUs ).

If we went down the route of cap'n proto we can rely on the embedded RPC and use that as a separate C2 protocol. I have used Cap'n Proto, for example, and we could easily define those RPC calls to lessen the load,  but I would likely change this ticket to exploring alternative protocols/wire formats and then making a decision as an addition to what we have now as opposed to a replacement.

We will be forced to maintain it but we have no choice either as we have users who specifically don't want protobuf and we have released it so we are forced to maintain it and improve its testing.

Keep in mind that whomever tackles this will need to balance maintaining what we already have ( not replacing ) so we then increase what we must test. We can rely on capn proto or protobuf's tests, but then we still need to perform our test in the venn diagram of testing. Would that effort be better spent improving tests for our work since we already need to do that and must maintain it?

I'll leave that question up to the implementor – but it's certainly not an affront against doing this work, more a word of caution. I'm glad we decided against protobuf at the time, though – would have made that decision again. It was a calculated risk with many thoughts in mind – but would love to see something else like protobuf/flatbuffers/capn .

Aside from this comment I would love the community to weigh in and make a choice. I've made my points known and have no intention of implementing this myself due to other technical debt I feel is more important so thoughts on what to do are up to the implementation seeker.


was (Author: phrocker):
[~bakaid]  I like some of the concepts you present but prior CVEs were why we moved away from protobuf initially in community discussions.  I think offloading the testing to protobuf maintainers is a reason to have Protobuf but I think both will need to exist.  We also considered flatbuffers and capn proto – would love to see that added to this discussion.

 

I think this ultimately would be either adding protobuf/flatbuffers/capn as a selectable line format. We have a very simple line format that isn't going to change ( this is because our line formats are designed to use the RESTFul API for the complicated stuff ). The line format is simply for sending a very tiny subset of information. We also want to use the same code for nanofi, where I think our consumers are more likely to appreciate not including protobuf. As a result we can add this as an extension ( in fact, I love the idea ) – especially since we can mitigate concerns of others by having it be configurable in the agent ( and compile time for nanofi ECUs ).

If we went down the route of cap'n proto we can rely on the embedded RPC and use that as a separate C2 protocol. I have used Cap'n Proto, for example, and we could easily define those RPC calls to lessen the load,  but I would likely change this ticket to exploring alternative protocols/wire formats and then making a decision as an addition to what we have now as opposed to a replacement.

We will be forced to maintain it but we have no choice either as we have users who specifically don't want protobuf and we have released it so we are forced to maintain it and improve its testing.

Keep in mind that whomever tackles this will need to balance maintaining what we already have ( not replacing ) so we then increase what we must test. We can rely on capn proto or protobuf's tests, but then we still need to perform our test in the venn diagram of testing. Would that effort be better spent improving tests for our work since we already need to do that and must maintain it?

I'll leave that question up to the implementor – but it's certainly not an affront against doing this work, more a word of caution. I'm glad we decided against protobuf at the time, though – would have made that decision again. It was a calculated risk with many thoughts in mind – but would love to see something else like protobuf/flatbuffers/capn .


Aside from this comment I would love the community to weigh in and make a choice. I've made my points known and have no intention of implementing this myself due to other technical debt I feel is more important so thoughts on what to do are up to the implementation seeker.

> Consider moving the C2 line protocol to Google Protobuf
> -------------------------------------------------------
>
>                 Key: MINIFICPP-954
>                 URL: https://issues.apache.org/jira/browse/MINIFICPP-954
>             Project: Apache NiFi MiNiFi C++
>          Issue Type: Improvement
>            Reporter: Daniel Bakai
>            Priority: Major
>
> * Very compact serialized format
>  * Easy protocol description
>  * Support for generating parsers for many languages from the proto file
>  * Has good versioning support
>  * Thoroughly tested, fuzzed
>  * Less chance of introducing security vulnerabilities with hand-written parsers



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)