You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by "Q1ngbo (via GitHub)" <gi...@apache.org> on 2026/01/15 11:35:41 UTC

[PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Q1ngbo opened a new pull request, #3197:
URL: https://github.com/apache/brpc/pull/3197

   ### What problem does this PR solve?
   
   Issue Number: #3062 #3196 
   
   Problem Summary:
   Hi, 本提交添加了对flatbuffers协议的完整支持。我增加了与“baidu_std”并列的协议,暂时命名为“fb_rpc”,主要实现在文件flatbuffers_protocol.cpp/h中。我认为以下2点值得关注:
   1. 类似于pb协议(即"baidu_std")的CallMethod,fb协议的请求起始于FBCallMethod,但二者十分相似,仅有几行差异,所以我使用函数模板CallMethodInternal提取了二者中的公共部分,使用参数is_pb区分,CallMethod传入参数为true,FBCallMethod为false。较宏观的benchmark测试显示修改后对pb协议几乎没有影响。
   2. Protocol类中的SerializeRequest和PackRequest这两个hook将形参固定为了google::protobuf类型,为了支持fb协议,我将它们改为了void*类型,并对所有协议中的这两个hook进行了适配,在对应函数开头加上了类型转换。
   
   由于该提交依赖于#3196,且编译机器上未安装flatbuffers,所以本pr暂时无法直接编译。我计划根据评审老师的意见再进行修改并添加单测等完整实现。
   
   ### Check List:
   - Please make sure your changes are compilable.
   - When providing us with a new feature, it is best to add related tests.
   - Please follow [Contributor Covenant Code of Conduct](https://github.com/apache/brpc/blob/master/CODE_OF_CONDUCT.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2936319837


##########
src/brpc/protocol.h:
##########
@@ -101,7 +101,7 @@ struct Protocol {
     typedef void (*SerializeRequest)(
         butil::IOBuf* request_buf,
         Controller* cntl,
-        const google::protobuf::Message* request);
+        const void* request_obj);

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "AnDiXL (via GitHub)" <gi...@apache.org>.
AnDiXL commented on PR #3197:
URL: https://github.com/apache/brpc/pull/3197#issuecomment-5757847800

   RPC, examples, ON CI and docs (including message prerequisites): [commits](https://github.com/AnDiXL/brpc/compare/2ce519a537955ac2a031db3bc5a2b60f8dd0e93a...c641abbc8cc3a512b264712fa852f92c2dcf75e0).
   
   Thanks @Spicy-cream for your interest in FlatBuffers and collaboration on LinQuickDev/brpc#45.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "chenBright (via GitHub) via dev" <de...@brpc.apache.org>.
chenBright commented on PR #3197:
URL: https://github.com/apache/brpc/pull/3197#issuecomment-6757839813

   FlatBuffers has been implemented in #3558 and #3559, so this PR is being closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "ivanallen (via GitHub)" <gi...@apache.org>.
ivanallen commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2707588137


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;

Review Comment:
   method_index 是怎么保证兼容性的。在另一个 pr 中,并没有看到为 method 声明 index,那么我猜可能是由 flatc 自动生成的。
   假如用户删除,或者增加 method,那么可能会导致对应的 method index 产生变化,在升级的时候会出现问题。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "ivanallen (via GitHub)" <gi...@apache.org>.
ivanallen commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2711489226


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;

Review Comment:
   > > method_index 是怎么保证兼容性的。在另一个 pr 中,并没有看到为 method 声明 index,那么我猜可能是由 flatc 自动生成的。 假如用户删除,或者增加 method,那么可能会导致对应的 method index 产生变化,在升级的时候会出现问题。
   > 
   > method的index是由flatc生成的,可以参考example/benchmark_fb/test.brpc.fb.cpp中的descriptor(),该函数中创建了一个BrpcDescriptorTable,其第三个参数即method_name_list保存了所有的方法名,初始化时会按序解析并为method赋予index。Server接收到请求后,会根据service_index与method_index查找method,如果查找失败则调用controller::SetFailed并设置ENOMETHOD。否则会调用test.brpc.fb.cpp中的FBCallMethod函数,在switch语句中根据method index来判断执行哪个method。
   > 
   > 如果server侧增加了新方法,且未修改method顺序,那么旧有的方法调用不会受影响,所以旧client仍能与新server通信;反之新client向旧server请求新方法时会产生“ENOMETHOD”错误。删除方法也类似,只要确保method index不受影响就能保证兼容性。
   
   看了下你的实现逻辑,看起来是按照顺序分配 index 的。比如 Method["Echo", "Write", "Read"],如果删除了 Write,那么 Read 的 index 就会从 2 变成 1. 所以这里用户还需要保证不能从中间删除 Method 才行吧。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "Q1ngbo (via GitHub)" <gi...@apache.org>.
Q1ngbo commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2711246886


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;
+        int32_t method_index;
+    } request;
+    int32_t message_size;
+    int32_t attachment_size;
+    int64_t correlation_id;
+}__attribute__((packed));
+
+struct FBRpcResponseMeta {
+    struct {
+        int32_t error_code;
+    } response;
+    int32_t message_size;
+    int32_t attachment_size;
+    int64_t correlation_id;
+}__attribute__((packed));
+
+struct FBRpcRequestHeader {
+    char header[12];
+    struct FBRpcRequestMeta meta;
+}__attribute__((packed));
+
+struct FBRpcResponseHeader {
+    char header[12];
+    struct FBRpcResponseMeta meta;
+}__attribute__((packed));
+
+bool inline ParseFbFromIOBuf(brpc::flatbuffers::Message* msg, size_t msg_size, const butil::IOBuf& buf) {
+    return brpc::flatbuffers::ParseFbFromIOBUF(msg, msg_size, buf);
+}
+
+// Notes:
+// 1. 12-byte header [BRPC][body_size][meta_size]
+// 2. body_size and meta_size are in network byte order
+// 3. Use service->service_index + method_index to specify the method to call
+// 4. `attachment_size' is set iff request/response has attachment
+// 5. Not supported: chunk_info
+
+// Pack header into `buf'
+
+static inline void PackFlatbuffersRpcHeader(char* rpc_header, int meta_size, int payload_size) {
+    // supress strict-aliasing warning.
+    uint32_t* dummy = (uint32_t*)rpc_header;
+    *dummy = *(uint32_t*)"BRPC";
+    butil::RawPacker(rpc_header + 4)
+        .pack32(meta_size + payload_size)
+        .pack32(meta_size);
+}
+
+static inline bool ParseMetaBufferFromIOBUF(butil::SingleIOBuf* dest,
+            const butil::IOBuf& source, uint32_t msg_size) {
+    return dest->assign(source, msg_size);
+}
+
+ParseResult ParseFlatBuffersMessage(butil::IOBuf* source, Socket* socket,
+                            bool /*read_eof*/, const void*) {
+    char header_buf[12];
+    const size_t n = source->copy_to(header_buf, sizeof(header_buf));
+    if (n >= 4) {
+        void* dummy = header_buf;
+        if (*(const uint32_t*)dummy != *(const uint32_t*)"BRPC") {
+            return MakeParseError(PARSE_ERROR_TRY_OTHERS);
+        }
+    } else {
+        if (memcmp(header_buf, "BRPC", n) != 0) {
+            return MakeParseError(PARSE_ERROR_TRY_OTHERS);
+        }
+    }
+    if (n < sizeof(header_buf)) {
+        return MakeParseError(PARSE_ERROR_NOT_ENOUGH_DATA);
+    }
+    uint32_t body_size;
+    uint32_t meta_size;
+    butil::RawUnpacker(header_buf + 4).unpack32(body_size).unpack32(meta_size);
+    if (body_size > FLAGS_max_body_size) {
+        // We need this log to report the body_size to give users some clues
+        // which is not printed in InputMessenger.
+        LOG(ERROR) << "body_size=" << body_size << " from "
+                   << socket->remote_side() << " is too large";
+        return MakeParseError(PARSE_ERROR_TOO_BIG_DATA);
+    } else if (source->length() < sizeof(header_buf) + body_size) {
+        return MakeParseError(PARSE_ERROR_NOT_ENOUGH_DATA);
+    }
+    if (meta_size > body_size) {
+        LOG(ERROR) << "meta_size=" << meta_size << " is bigger than body_size="
+                   << body_size;
+        // Pop the message
+        source->pop_front(sizeof(header_buf) + body_size);
+        return MakeParseError(PARSE_ERROR_TRY_OTHERS);
+    }
+    source->pop_front(sizeof(header_buf));
+    MostCommonMessage* msg = MostCommonMessage::Get();
+    source->cutn(&msg->meta, meta_size);
+    source->cutn(&msg->payload, body_size - meta_size);
+    return MakeMessage(msg);
+}
+
+static void SendFlatBuffersRpcResponse(int64_t correlation_id,
+                     Controller* cntl,
+                     brpc::flatbuffers::Message* req,
+                     brpc::flatbuffers::Message* res,
+                     const Server* server,
+                     MethodStatus* method_status_raw,
+                     int64_t received_us) {
+    ControllerPrivateAccessor accessor(cntl);
+    Socket* sock = accessor.get_sending_socket();
+    ConcurrencyRemover concurrency_remover(method_status_raw, cntl, received_us);
+    std::unique_ptr<Controller, LogErrorTextAndDelete> recycle_cntl(cntl);
+    std::unique_ptr<brpc::flatbuffers::Message> recycle_req(req);
+    std::unique_ptr<brpc::flatbuffers::Message> recycle_res(res);
+    //ScopedRemoveConcurrency remove_concurrency_dummy(server, cntl);
+    if (cntl->IsCloseConnection()) {
+        sock->SetFailed();
+        return;
+    }
+    bool append_body = false;
+    butil::IOBuf res_body;
+    // `res' can be NULL here, in which case we don't serialize it
+    // If user calls `SetFailed' on Controller, we don't serialize
+    // response either
+    struct FBRpcResponseHeader *rpc_header = NULL;
+    uint32_t reserve_size = sizeof(struct FBRpcResponseHeader);
+
+    if (res != NULL && !cntl->Failed()) {
+        rpc_header = static_cast<struct FBRpcResponseHeader*>(res->reduce_meta_size_and_get_buf(sizeof(struct FBRpcResponseHeader)));
+        if (BAIDU_UNLIKELY(rpc_header == NULL)) {
+            cntl->SetFailed(ERESPONSE, "Fail to reduce meta size and get buf");
+        } else {
+            if (!brpc::flatbuffers::SerializeFbToIOBUF(res, res_body)) {
+                cntl->SetFailed(ERESPONSE, "Fail to serialize response");
+            } else {
+                append_body = true;
+            }
+        }
+    }
+
+    // Don't use res->ByteSize() since it may be compressed
+    size_t res_size = 0;
+    size_t attached_size = 0;
+    size_t meta_size = sizeof(struct FBRpcResponseMeta);
+    if (append_body && rpc_header != NULL) {
+        res_size = res_body.length() - reserve_size;
+        attached_size = cntl->response_attachment().length();
+        PackFlatbuffersRpcHeader(rpc_header->header,
+                    meta_size, res_size + attached_size);
+        rpc_header->meta.message_size = res_size;
+        rpc_header->meta.attachment_size = attached_size;
+        rpc_header->meta.response.error_code = cntl->ErrorCode();
+        rpc_header->meta.correlation_id = correlation_id;
+        if (attached_size > 0) {
+            res_body.append(cntl->response_attachment().movable());
+        }
+    } else {    // error response
+        struct FBRpcResponseHeader tmp_header;
+        tmp_header.meta.message_size = 0;
+        tmp_header.meta.attachment_size = 0;
+        tmp_header.meta.response.error_code = cntl->ErrorCode();
+        tmp_header.meta.correlation_id = correlation_id;
+        PackFlatbuffersRpcHeader(tmp_header.header,
+                                        meta_size, 0);
+        res_body.clear();
+        res_body.append((void const*) &tmp_header,
+            sizeof(struct FBRpcResponseHeader));
+    }
+    Socket::WriteOptions wopt;
+    wopt.ignore_eovercrowded = true;
+    if (sock->Write(&res_body, &wopt) != 0) {
+        const int errcode = errno;
+        PLOG_IF(WARNING, errcode != EPIPE) << "Fail to write into " << *sock;
+        cntl->SetFailed(errcode, "Fail to write into %s",
+                        sock->description().c_str());
+        return;
+    }
+}
+
+void ProcessFlatBuffersRequest(InputMessageBase* msg_base) {
+    DestroyingPtr<MostCommonMessage> msg(static_cast<MostCommonMessage*>(msg_base));
+    SocketUniquePtr socket_guard(msg->ReleaseSocket());
+    Socket* socket = socket_guard.get();
+    const Server* server = static_cast<const Server*>(msg_base->arg());
+    ScopedNonServiceError non_service_error(server);
+    butil::SingleIOBuf meta_buf;
+    if (!ParseMetaBufferFromIOBUF(&meta_buf,
+            msg->meta, sizeof(struct FBRpcRequestMeta))) {
+        LOG(WARNING) << "Fail to parse RpcMeta from " << *socket;
+        socket->SetFailed(EREQUEST, "Fail to parse RpcMeta from %s",
+                                        socket->description().c_str());
+        return;
+    }
+    const struct FBRpcRequestMeta* meta =
+        static_cast<const struct FBRpcRequestMeta*>(meta_buf.get_begin());
+    if (!meta) {
+        LOG(WARNING) << "RpcMeta from " << *socket << " is NULL";
+        socket->SetFailed(EREQUEST, "Fail to parse RpcMeta from %s",
+                                        socket->description().c_str());
+        return;
+    }
+
+    std::unique_ptr<Controller> cntl;
+    cntl.reset(new (std::nothrow) Controller);
+    if (NULL == cntl.get()) {
+        LOG(WARNING) << "Fail to new Controller";
+        return;
+    }
+
+    std::unique_ptr<brpc::flatbuffers::Message> req;
+    std::unique_ptr<brpc::flatbuffers::Message> res;
+
+    ServerPrivateAccessor server_accessor(server);
+
+    ControllerPrivateAccessor accessor(cntl.get());
+    accessor.set_server(server)
+        .set_peer_id(socket->id())
+        .set_remote_side(socket->remote_side())
+        .set_local_side(socket->local_side())
+        .set_request_protocol(PROTOCOL_FLATBUFFERS_RPC)
+        .move_in_server_receiving_sock(socket_guard);
+    MethodStatus* method_status = NULL;
+    do {
+        if (!server->IsRunning()) {
+            cntl->SetFailed(ELOGOFF, "Server is stopping");
+            break;
+        }
+
+        if (socket->is_overcrowded()) {
+            cntl->SetFailed(EOVERCROWDED, "Connection to %s is overcrowded",
+                            butil::endpoint2str(socket->remote_side()).c_str());
+            break;
+        }
+
+        if (!server_accessor.AddConcurrency(cntl.get())) {
+            cntl->SetFailed(ELIMIT, "Reached server's max_concurrency=%d",
+                            server->options().max_concurrency);
+            break;
+        }
+
+        const Server::FlatBuffersMethodProperty* mp =
+            server_accessor.FindFlatBufferMethodPropertyByIndex(meta->request.service_index,
+                                                                        meta->request.method_index);
+        if (NULL == mp) {
+            cntl->SetFailed(ENOMETHOD, "Fail to find method_index=%d service_index=%u ",
+                                                                meta->request.method_index,
+                                                                meta->request.service_index);
+            break;
+        }
+        // Switch to service-specific error.
+        non_service_error.release();
+        if (mp->status) {
+            method_status = mp->status;
+            if (!method_status->OnRequested()) {
+                cntl->SetFailed(ELIMIT, "Reached %s's MaxConcurrency=%d",
+                                mp->method->full_name().c_str(),
+                                method_status->MaxConcurrency());
+                break;
+            }
+        }
+        brpc::flatbuffers::Service* svc = mp->service;
+        const brpc::flatbuffers::MethodDescriptor* method = mp->method;
+        accessor.set_fb_method(method);
+        const int reqsize = static_cast<int>(msg->payload.size());
+        butil::IOBuf req_buf;
+        butil::IOBuf* req_buf_ptr = &msg->payload;
+        if (meta->attachment_size > 0) {
+            if (reqsize < meta->attachment_size) {
+                cntl->SetFailed(EREQUEST,
+                    "attachment_size=%d is larger than request_size=%d",
+                     meta->attachment_size, reqsize);
+                break;
+            }
+            int body_without_attachment_size = reqsize - meta->attachment_size;
+            msg->payload.cutn(&req_buf, body_without_attachment_size);
+            req_buf_ptr = &req_buf;
+            cntl->request_attachment().swap(msg->payload);
+        }
+
+        req.reset(new brpc::flatbuffers::Message());
+        if (!brpc::flatbuffers::ParseFbFromIOBUF(req.get(), meta->message_size, *req_buf_ptr)) {
+            cntl->SetFailed(EREQUEST, "Fail to parse request message, "
+                            "request_size=%d", reqsize);
+            break;
+        }
+        res.reset(new brpc::flatbuffers::Message());
+        // `socket' will be held until response has been sent
+        google::protobuf::Closure* done = ::brpc::NewCallback<
+            int64_t, Controller*, brpc::flatbuffers::Message*,
+            brpc::flatbuffers::Message*, const Server*,
+            MethodStatus*, int64_t>(
+                &SendFlatBuffersRpcResponse, meta->correlation_id, cntl.get(),
+                req.get(), res.get(), server,
+                method_status, msg->received_us());
+
+        // optional, just release resourse ASAP
+        msg.reset();
+        req_buf.clear();
+        //only used in polling thread

Review Comment:
   > 这行注释里说的 polling thread 是指什么?
   
   无意义内容,将被删除



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r3051081990


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {

Review Comment:
   至少在协议中要加上meta的长度,这样以后能增加字段吧。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "AnDiXL (via GitHub)" <gi...@apache.org>.
AnDiXL commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2731527931


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;

Review Comment:
   hi,请问这个提交还有在看不,还有什么需要我们改进的吗 @ivanallen 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2749177327


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {

Review Comment:
   有没有考虑过这个Meta也用flatbuffers格式呢?这样以后新增字段不会破坏兼容性



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "ivanallen (via GitHub)" <gi...@apache.org>.
ivanallen commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2736401888


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;

Review Comment:
   > hi,请问这个提交还有在看不,还有什么需要我们改进的吗 @ivanallen
   
   我周末再看下哈。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "chenBright (via GitHub) via dev" <de...@brpc.apache.org>.
chenBright commented on PR #3197:
URL: https://github.com/apache/brpc/pull/3197#issuecomment-6757785708

   @AnDiXL PTAL. FlatBuffers CI occasionally fails. https://github.com/apache/brpc/actions/runs/42735197577/job/129562607455


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "Q1ngbo (via GitHub)" <gi...@apache.org>.
Q1ngbo commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2711244192


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;

Review Comment:
   > method_index 是怎么保证兼容性的。在另一个 pr 中,并没有看到为 method 声明 index,那么我猜可能是由 flatc 自动生成的。 假如用户删除,或者增加 method,那么可能会导致对应的 method index 产生变化,在升级的时候会出现问题。
   
   method的index是由flatc生成的,可以参考example/benchmark_fb/test.brpc.fb.cpp中的descriptor(),该函数中创建了一个BrpcDescriptorTable,其第三个参数即method_name_list保存了所有的方法名,初始化时会按序解析并为method赋予index。Server接收到请求后,会根据service_index与method_index查找method,如果查找失败则调用controller::SetFailed并设置ENOMETHOD。否则会调用test.brpc.fb.cpp中的FBCallMethod函数,在switch语句中根据method index来判断执行哪个method。
   
   如果server侧增加了新方法,且未修改method顺序,那么旧有的方法调用不会受影响,所以旧client仍能与新server通信;反之新client向旧server请求新方法时会产生“ENOMETHOD”错误。删除方法也类似,只要确保method index不受影响就能保证兼容性。
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "chenBright (via GitHub) via dev" <de...@brpc.apache.org>.
chenBright closed pull request #3197: flatbuffers support(3/3): Add flatbuffers protocol support for brpc
URL: https://github.com/apache/brpc/pull/3197


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "ivanallen (via GitHub)" <gi...@apache.org>.
ivanallen commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2707588137


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;

Review Comment:
   service_index 是怎么保证兼容性的。在另一个 pr 中,并没有看到为 method 声明 index,那么我猜可能是由 flatc 自动生成的。
   假如用户删除,或者增加 method,那么可能会导致对应的 method index 产生变化,在升级的时候会出现问题。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2749164098


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;
+        int32_t method_index;
+    } request;
+    int32_t message_size;
+    int32_t attachment_size;
+    int64_t correlation_id;
+}__attribute__((packed));
+
+struct FBRpcResponseMeta {
+    struct {
+        int32_t error_code;
+    } response;
+    int32_t message_size;
+    int32_t attachment_size;
+    int64_t correlation_id;
+}__attribute__((packed));
+
+struct FBRpcRequestHeader {
+    char header[12];
+    struct FBRpcRequestMeta meta;
+}__attribute__((packed));
+
+struct FBRpcResponseHeader {
+    char header[12];
+    struct FBRpcResponseMeta meta;
+}__attribute__((packed));
+
+bool inline ParseFbFromIOBuf(brpc::flatbuffers::Message* msg, size_t msg_size, const butil::IOBuf& buf) {
+    return brpc::flatbuffers::ParseFbFromIOBUF(msg, msg_size, buf);
+}
+
+// Notes:
+// 1. 12-byte header [BRPC][body_size][meta_size]
+// 2. body_size and meta_size are in network byte order
+// 3. Use service->service_index + method_index to specify the method to call
+// 4. `attachment_size' is set iff request/response has attachment
+// 5. Not supported: chunk_info
+
+// Pack header into `buf'
+
+static inline void PackFlatbuffersRpcHeader(char* rpc_header, int meta_size, int payload_size) {
+    // supress strict-aliasing warning.
+    uint32_t* dummy = (uint32_t*)rpc_header;
+    *dummy = *(uint32_t*)"BRPC";
+    butil::RawPacker(rpc_header + 4)
+        .pack32(meta_size + payload_size)
+        .pack32(meta_size);
+}
+
+static inline bool ParseMetaBufferFromIOBUF(butil::SingleIOBuf* dest,
+            const butil::IOBuf& source, uint32_t msg_size) {
+    return dest->assign(source, msg_size);
+}
+
+ParseResult ParseFlatBuffersMessage(butil::IOBuf* source, Socket* socket,
+                            bool /*read_eof*/, const void*) {
+    char header_buf[12];
+    const size_t n = source->copy_to(header_buf, sizeof(header_buf));
+    if (n >= 4) {
+        void* dummy = header_buf;
+        if (*(const uint32_t*)dummy != *(const uint32_t*)"BRPC") {
+            return MakeParseError(PARSE_ERROR_TRY_OTHERS);
+        }
+    } else {
+        if (memcmp(header_buf, "BRPC", n) != 0) {

Review Comment:
   BRPC这个名字改一下,比如FRPC?



##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;
+        int32_t method_index;
+    } request;
+    int32_t message_size;
+    int32_t attachment_size;
+    int64_t correlation_id;
+}__attribute__((packed));
+
+struct FBRpcResponseMeta {
+    struct {
+        int32_t error_code;
+    } response;
+    int32_t message_size;
+    int32_t attachment_size;
+    int64_t correlation_id;
+}__attribute__((packed));
+
+struct FBRpcRequestHeader {
+    char header[12];
+    struct FBRpcRequestMeta meta;
+}__attribute__((packed));
+
+struct FBRpcResponseHeader {
+    char header[12];
+    struct FBRpcResponseMeta meta;
+}__attribute__((packed));
+
+bool inline ParseFbFromIOBuf(brpc::flatbuffers::Message* msg, size_t msg_size, const butil::IOBuf& buf) {
+    return brpc::flatbuffers::ParseFbFromIOBUF(msg, msg_size, buf);
+}
+
+// Notes:
+// 1. 12-byte header [BRPC][body_size][meta_size]
+// 2. body_size and meta_size are in network byte order
+// 3. Use service->service_index + method_index to specify the method to call
+// 4. `attachment_size' is set iff request/response has attachment
+// 5. Not supported: chunk_info
+
+// Pack header into `buf'
+
+static inline void PackFlatbuffersRpcHeader(char* rpc_header, int meta_size, int payload_size) {
+    // supress strict-aliasing warning.
+    uint32_t* dummy = (uint32_t*)rpc_header;
+    *dummy = *(uint32_t*)"BRPC";
+    butil::RawPacker(rpc_header + 4)
+        .pack32(meta_size + payload_size)
+        .pack32(meta_size);
+}
+
+static inline bool ParseMetaBufferFromIOBUF(butil::SingleIOBuf* dest,
+            const butil::IOBuf& source, uint32_t msg_size) {
+    return dest->assign(source, msg_size);
+}
+
+ParseResult ParseFlatBuffersMessage(butil::IOBuf* source, Socket* socket,
+                            bool /*read_eof*/, const void*) {
+    char header_buf[12];
+    const size_t n = source->copy_to(header_buf, sizeof(header_buf));
+    if (n >= 4) {
+        void* dummy = header_buf;
+        if (*(const uint32_t*)dummy != *(const uint32_t*)"BRPC") {
+            return MakeParseError(PARSE_ERROR_TRY_OTHERS);
+        }
+    } else {
+        if (memcmp(header_buf, "BRPC", n) != 0) {
+            return MakeParseError(PARSE_ERROR_TRY_OTHERS);
+        }
+    }
+    if (n < sizeof(header_buf)) {

Review Comment:
   适当加一些空行,代码看上去太挤了



##########
src/brpc/protocol.h:
##########
@@ -101,7 +101,7 @@ struct Protocol {
     typedef void (*SerializeRequest)(
         butil::IOBuf* request_buf,
         Controller* cntl,
-        const google::protobuf::Message* request);
+        const void* request_obj);

Review Comment:
   这个改动影响太大了,我认为是不可接受的。不仅是brpc代码库中的所有protocol实现需要修改,而且其他用户自己实现的protocol也要修改(即不兼容升级)
   我提一个思路你参考一下:你可以让brpc::flatbuffers::Message继承brpc::NonreflectableMessage,这样这个message就自动继承了google::protobuf::Message,这样就可以在不修改brpc框架接口的前提下扩展出flatbuffer协议。你可以参考brpc Redis/Memcached等协议的实现,这些协议实际上并不用到protobuf,但也保持了google::protobuf::Message的接口形式。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "Q1ngbo (via GitHub)" <gi...@apache.org>.
Q1ngbo commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r3050557196


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {

Review Comment:
   
   不太好实现。因为改用fbs定义meta部分后,需要通过FlatBufferBuilder构建meta,赋值时需要走flatbuffers的接口,这会有额外的处理开销。出于性能考虑,当前Meta字段与rpc message被放在了连续的内存中,使用FlatBufferBuilder创建meta时就无法使用该优化了。
   也无法直接使用flatc生成的meta struct,因为flatc会将成员变量属性设置为private,且只能通过特定接口访问成员变量。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "Q1ngbo (via GitHub)" <gi...@apache.org>.
Q1ngbo commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2712284698


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;

Review Comment:
   是的



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "ivanallen (via GitHub)" <gi...@apache.org>.
ivanallen commented on code in PR #3197:
URL: https://github.com/apache/brpc/pull/3197#discussion_r2707555623


##########
src/brpc/policy/flatbuffers_protocol.cpp:
##########
@@ -0,0 +1,472 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "butil/logging.h"                       // LOG()
+#include "butil/iobuf.h"                         // butil::IOBuf
+#include "butil/single_iobuf.h"                  // butil::SingleIOBuf
+#include "butil/time.h"
+
+#include "butil/raw_pack.h"                      // RawPacker RawUnpacker
+
+#include "brpc/controller.h"                     // Controller
+#include "brpc/socket.h"                         // Socket
+#include "brpc/server.h"                         // Server
+#include "brpc/stream_impl.h"
+#include "brpc/rpc_dump.h"                       // SampledRequest
+#include "brpc/policy/most_common_message.h"
+#include "brpc/details/controller_private_accessor.h"
+#include "brpc/details/server_private_accessor.h"
+#include "brpc/policy/flatbuffers_protocol.h"
+
+namespace brpc {
+namespace policy {
+
+struct FBRpcRequestMeta {
+    struct {
+        uint32_t service_index;
+        int32_t method_index;
+    } request;
+    int32_t message_size;
+    int32_t attachment_size;
+    int64_t correlation_id;
+}__attribute__((packed));
+
+struct FBRpcResponseMeta {
+    struct {
+        int32_t error_code;
+    } response;
+    int32_t message_size;
+    int32_t attachment_size;
+    int64_t correlation_id;
+}__attribute__((packed));
+
+struct FBRpcRequestHeader {
+    char header[12];
+    struct FBRpcRequestMeta meta;
+}__attribute__((packed));
+
+struct FBRpcResponseHeader {
+    char header[12];
+    struct FBRpcResponseMeta meta;
+}__attribute__((packed));
+
+bool inline ParseFbFromIOBuf(brpc::flatbuffers::Message* msg, size_t msg_size, const butil::IOBuf& buf) {
+    return brpc::flatbuffers::ParseFbFromIOBUF(msg, msg_size, buf);
+}
+
+// Notes:
+// 1. 12-byte header [BRPC][body_size][meta_size]
+// 2. body_size and meta_size are in network byte order
+// 3. Use service->service_index + method_index to specify the method to call
+// 4. `attachment_size' is set iff request/response has attachment
+// 5. Not supported: chunk_info
+
+// Pack header into `buf'
+
+static inline void PackFlatbuffersRpcHeader(char* rpc_header, int meta_size, int payload_size) {
+    // supress strict-aliasing warning.
+    uint32_t* dummy = (uint32_t*)rpc_header;
+    *dummy = *(uint32_t*)"BRPC";
+    butil::RawPacker(rpc_header + 4)
+        .pack32(meta_size + payload_size)
+        .pack32(meta_size);
+}
+
+static inline bool ParseMetaBufferFromIOBUF(butil::SingleIOBuf* dest,
+            const butil::IOBuf& source, uint32_t msg_size) {
+    return dest->assign(source, msg_size);
+}
+
+ParseResult ParseFlatBuffersMessage(butil::IOBuf* source, Socket* socket,
+                            bool /*read_eof*/, const void*) {
+    char header_buf[12];
+    const size_t n = source->copy_to(header_buf, sizeof(header_buf));
+    if (n >= 4) {
+        void* dummy = header_buf;
+        if (*(const uint32_t*)dummy != *(const uint32_t*)"BRPC") {
+            return MakeParseError(PARSE_ERROR_TRY_OTHERS);
+        }
+    } else {
+        if (memcmp(header_buf, "BRPC", n) != 0) {
+            return MakeParseError(PARSE_ERROR_TRY_OTHERS);
+        }
+    }
+    if (n < sizeof(header_buf)) {
+        return MakeParseError(PARSE_ERROR_NOT_ENOUGH_DATA);
+    }
+    uint32_t body_size;
+    uint32_t meta_size;
+    butil::RawUnpacker(header_buf + 4).unpack32(body_size).unpack32(meta_size);
+    if (body_size > FLAGS_max_body_size) {
+        // We need this log to report the body_size to give users some clues
+        // which is not printed in InputMessenger.
+        LOG(ERROR) << "body_size=" << body_size << " from "
+                   << socket->remote_side() << " is too large";
+        return MakeParseError(PARSE_ERROR_TOO_BIG_DATA);
+    } else if (source->length() < sizeof(header_buf) + body_size) {
+        return MakeParseError(PARSE_ERROR_NOT_ENOUGH_DATA);
+    }
+    if (meta_size > body_size) {
+        LOG(ERROR) << "meta_size=" << meta_size << " is bigger than body_size="
+                   << body_size;
+        // Pop the message
+        source->pop_front(sizeof(header_buf) + body_size);
+        return MakeParseError(PARSE_ERROR_TRY_OTHERS);
+    }
+    source->pop_front(sizeof(header_buf));
+    MostCommonMessage* msg = MostCommonMessage::Get();
+    source->cutn(&msg->meta, meta_size);
+    source->cutn(&msg->payload, body_size - meta_size);
+    return MakeMessage(msg);
+}
+
+static void SendFlatBuffersRpcResponse(int64_t correlation_id,
+                     Controller* cntl,
+                     brpc::flatbuffers::Message* req,
+                     brpc::flatbuffers::Message* res,
+                     const Server* server,
+                     MethodStatus* method_status_raw,
+                     int64_t received_us) {
+    ControllerPrivateAccessor accessor(cntl);
+    Socket* sock = accessor.get_sending_socket();
+    ConcurrencyRemover concurrency_remover(method_status_raw, cntl, received_us);
+    std::unique_ptr<Controller, LogErrorTextAndDelete> recycle_cntl(cntl);
+    std::unique_ptr<brpc::flatbuffers::Message> recycle_req(req);
+    std::unique_ptr<brpc::flatbuffers::Message> recycle_res(res);
+    //ScopedRemoveConcurrency remove_concurrency_dummy(server, cntl);
+    if (cntl->IsCloseConnection()) {
+        sock->SetFailed();
+        return;
+    }
+    bool append_body = false;
+    butil::IOBuf res_body;
+    // `res' can be NULL here, in which case we don't serialize it
+    // If user calls `SetFailed' on Controller, we don't serialize
+    // response either
+    struct FBRpcResponseHeader *rpc_header = NULL;
+    uint32_t reserve_size = sizeof(struct FBRpcResponseHeader);
+
+    if (res != NULL && !cntl->Failed()) {
+        rpc_header = static_cast<struct FBRpcResponseHeader*>(res->reduce_meta_size_and_get_buf(sizeof(struct FBRpcResponseHeader)));
+        if (BAIDU_UNLIKELY(rpc_header == NULL)) {
+            cntl->SetFailed(ERESPONSE, "Fail to reduce meta size and get buf");
+        } else {
+            if (!brpc::flatbuffers::SerializeFbToIOBUF(res, res_body)) {
+                cntl->SetFailed(ERESPONSE, "Fail to serialize response");
+            } else {
+                append_body = true;
+            }
+        }
+    }
+
+    // Don't use res->ByteSize() since it may be compressed
+    size_t res_size = 0;
+    size_t attached_size = 0;
+    size_t meta_size = sizeof(struct FBRpcResponseMeta);
+    if (append_body && rpc_header != NULL) {
+        res_size = res_body.length() - reserve_size;
+        attached_size = cntl->response_attachment().length();
+        PackFlatbuffersRpcHeader(rpc_header->header,
+                    meta_size, res_size + attached_size);
+        rpc_header->meta.message_size = res_size;
+        rpc_header->meta.attachment_size = attached_size;
+        rpc_header->meta.response.error_code = cntl->ErrorCode();
+        rpc_header->meta.correlation_id = correlation_id;
+        if (attached_size > 0) {
+            res_body.append(cntl->response_attachment().movable());
+        }
+    } else {    // error response
+        struct FBRpcResponseHeader tmp_header;
+        tmp_header.meta.message_size = 0;
+        tmp_header.meta.attachment_size = 0;
+        tmp_header.meta.response.error_code = cntl->ErrorCode();
+        tmp_header.meta.correlation_id = correlation_id;
+        PackFlatbuffersRpcHeader(tmp_header.header,
+                                        meta_size, 0);
+        res_body.clear();
+        res_body.append((void const*) &tmp_header,
+            sizeof(struct FBRpcResponseHeader));
+    }
+    Socket::WriteOptions wopt;
+    wopt.ignore_eovercrowded = true;
+    if (sock->Write(&res_body, &wopt) != 0) {
+        const int errcode = errno;
+        PLOG_IF(WARNING, errcode != EPIPE) << "Fail to write into " << *sock;
+        cntl->SetFailed(errcode, "Fail to write into %s",
+                        sock->description().c_str());
+        return;
+    }
+}
+
+void ProcessFlatBuffersRequest(InputMessageBase* msg_base) {
+    DestroyingPtr<MostCommonMessage> msg(static_cast<MostCommonMessage*>(msg_base));
+    SocketUniquePtr socket_guard(msg->ReleaseSocket());
+    Socket* socket = socket_guard.get();
+    const Server* server = static_cast<const Server*>(msg_base->arg());
+    ScopedNonServiceError non_service_error(server);
+    butil::SingleIOBuf meta_buf;
+    if (!ParseMetaBufferFromIOBUF(&meta_buf,
+            msg->meta, sizeof(struct FBRpcRequestMeta))) {
+        LOG(WARNING) << "Fail to parse RpcMeta from " << *socket;
+        socket->SetFailed(EREQUEST, "Fail to parse RpcMeta from %s",
+                                        socket->description().c_str());
+        return;
+    }
+    const struct FBRpcRequestMeta* meta =
+        static_cast<const struct FBRpcRequestMeta*>(meta_buf.get_begin());
+    if (!meta) {
+        LOG(WARNING) << "RpcMeta from " << *socket << " is NULL";
+        socket->SetFailed(EREQUEST, "Fail to parse RpcMeta from %s",
+                                        socket->description().c_str());
+        return;
+    }
+
+    std::unique_ptr<Controller> cntl;
+    cntl.reset(new (std::nothrow) Controller);
+    if (NULL == cntl.get()) {
+        LOG(WARNING) << "Fail to new Controller";
+        return;
+    }
+
+    std::unique_ptr<brpc::flatbuffers::Message> req;
+    std::unique_ptr<brpc::flatbuffers::Message> res;
+
+    ServerPrivateAccessor server_accessor(server);
+
+    ControllerPrivateAccessor accessor(cntl.get());
+    accessor.set_server(server)
+        .set_peer_id(socket->id())
+        .set_remote_side(socket->remote_side())
+        .set_local_side(socket->local_side())
+        .set_request_protocol(PROTOCOL_FLATBUFFERS_RPC)
+        .move_in_server_receiving_sock(socket_guard);
+    MethodStatus* method_status = NULL;
+    do {
+        if (!server->IsRunning()) {
+            cntl->SetFailed(ELOGOFF, "Server is stopping");
+            break;
+        }
+
+        if (socket->is_overcrowded()) {
+            cntl->SetFailed(EOVERCROWDED, "Connection to %s is overcrowded",
+                            butil::endpoint2str(socket->remote_side()).c_str());
+            break;
+        }
+
+        if (!server_accessor.AddConcurrency(cntl.get())) {
+            cntl->SetFailed(ELIMIT, "Reached server's max_concurrency=%d",
+                            server->options().max_concurrency);
+            break;
+        }
+
+        const Server::FlatBuffersMethodProperty* mp =
+            server_accessor.FindFlatBufferMethodPropertyByIndex(meta->request.service_index,
+                                                                        meta->request.method_index);
+        if (NULL == mp) {
+            cntl->SetFailed(ENOMETHOD, "Fail to find method_index=%d service_index=%u ",
+                                                                meta->request.method_index,
+                                                                meta->request.service_index);
+            break;
+        }
+        // Switch to service-specific error.
+        non_service_error.release();
+        if (mp->status) {
+            method_status = mp->status;
+            if (!method_status->OnRequested()) {
+                cntl->SetFailed(ELIMIT, "Reached %s's MaxConcurrency=%d",
+                                mp->method->full_name().c_str(),
+                                method_status->MaxConcurrency());
+                break;
+            }
+        }
+        brpc::flatbuffers::Service* svc = mp->service;
+        const brpc::flatbuffers::MethodDescriptor* method = mp->method;
+        accessor.set_fb_method(method);
+        const int reqsize = static_cast<int>(msg->payload.size());
+        butil::IOBuf req_buf;
+        butil::IOBuf* req_buf_ptr = &msg->payload;
+        if (meta->attachment_size > 0) {
+            if (reqsize < meta->attachment_size) {
+                cntl->SetFailed(EREQUEST,
+                    "attachment_size=%d is larger than request_size=%d",
+                     meta->attachment_size, reqsize);
+                break;
+            }
+            int body_without_attachment_size = reqsize - meta->attachment_size;
+            msg->payload.cutn(&req_buf, body_without_attachment_size);
+            req_buf_ptr = &req_buf;
+            cntl->request_attachment().swap(msg->payload);
+        }
+
+        req.reset(new brpc::flatbuffers::Message());
+        if (!brpc::flatbuffers::ParseFbFromIOBUF(req.get(), meta->message_size, *req_buf_ptr)) {
+            cntl->SetFailed(EREQUEST, "Fail to parse request message, "
+                            "request_size=%d", reqsize);
+            break;
+        }
+        res.reset(new brpc::flatbuffers::Message());
+        // `socket' will be held until response has been sent
+        google::protobuf::Closure* done = ::brpc::NewCallback<
+            int64_t, Controller*, brpc::flatbuffers::Message*,
+            brpc::flatbuffers::Message*, const Server*,
+            MethodStatus*, int64_t>(
+                &SendFlatBuffersRpcResponse, meta->correlation_id, cntl.get(),
+                req.get(), res.get(), server,
+                method_status, msg->received_us());
+
+        // optional, just release resourse ASAP
+        msg.reset();
+        req_buf.clear();
+        //only used in polling thread

Review Comment:
   这行注释里说的 polling thread 是指什么?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] flatbuffers support(3/3): Add flatbuffers protocol support for brpc (brpc)

Posted by "Q1ngbo (via GitHub)" <gi...@apache.org>.
Q1ngbo commented on PR #3197:
URL: https://github.com/apache/brpc/pull/3197#issuecomment-3754426995

   ### 补充内容
   #### fb协议处理的简要说明
   
   Client应用会创建Stub,然后调用fbs中指定的method方法,之后会去调用channel->FBCallMethod。这些方法都是由flatbuffers自动生成的,而FBCallMethod是在brpc中定义的,此处是二者间的一个桥梁。由于FBCallMethod的实现与针对pb协议的CallMethod非常类似,所以我使用函数模板CallMethodInternal提取了二者中的公共部分,使用参数is_pb区分,CallMethod传入参数为true,FBCallMethod为false。发送路径上的序列化(SerializeFlatBuffersRequest)、Pack(PackFlatBuffersRequest)等操作已经被定义在了Protocol fb_protocol中。
   
   序列化函数SerializeFlatBuffersRequest的执行逻辑非常简单,首先根据FBRpcRequestHeader大小来调整message前面预留的header buffer空间,然后零拷贝地将message中的SingleIOBuf append至目的IOBuf就结束了。值得注意的是,fb协议中header的各字段被硬编码了,不能像pb一样通过proto文件(baidu_rpc_meta.proto)调整。
    
   函数PackFlatBuffersRequest的功能与baidu_std协议中的PackRpcRequest函数类似,只不过rpc header部分已经预先分配好。
    
   Server应创建一个类,继承fb.h中生成的Service,并重载其virtual方法。在启动Server进程时,应将该类对象通过AddService将该service注册至brpc。接收到的请求会交由ProcessFlatBuffersRequest处理,它首先从msg->meta中提取出Header(这一步由SingleIOBuf::assign完成,可能会有拷贝);然后配置Controller,从msg->payload中解析出fb message(同样会调用SingleIOBuf::assign);交由应用响应。应用逻辑处理完成后,会通过SendFlatBuffersRpcResponse向Client发送response message,这里同样会将rpc header + message零拷贝地append到待发送IOBuf中。
   
   #### 性能测试
   我基于multi_threaded_echo_c++改造了benchmark程序,分别基于pb和fb协议实现,benchmark_fb的实现在#3196中。该程序基于bthread模型,共享同一个channel;每次发送时创建message,该message包含32B固定字段+ string组成的浮动字段(由request_size指定),proto文件定义如下;采用同步发送;输出指标包括延迟和QPS。
   ```c++
   syntax="proto2";
   option cc_generic_services = true;
   
   package test;
   
   message BenchmarkRequest {
       required int32 opcode = 1;
       required int32 echo_attachment = 2;
       required int64 attachment_size = 3;
       required int64 request_id = 4;
       optional int64 reserved = 5;
       optional string message = 4;
   };
   
   message BenchmarkResponse {
       required int32 opcode = 1;
       required int32 echo_attachment = 2;
       required int64 attachment_size = 3;
       required int64 request_id = 4;
       optional int64 reserved = 5;
       optional string message = 4;
   };
   
   service BenchmarkService {
         rpc Test(BenchmarkRequest) returns (BenchmarkResponse);
   };
   ```
   
   测试目的是为了说明fb的处理效率优于pb。为了避免网络影响,测试在同机上进行,分别为client和server绑定4个核。
   下表所示数据为thread_num=1,即bthread数目为1时的结果。测试过程中使用Req Size变化请求中的string长度来改变message总大小,由于message中包含32B的固定整数部分,所以message实际总大小为32B(固定部分) + Req Size,attachment为0。测试程序执行20s,记录平均QPS,单位为千次/秒。表格中pb(old)为原CallMethod实现,pb(new)为当前pr中基于函数模板CallMethodInternal的实现。表中最后一行展示的是fb相对于pb(new)的提升百分比,从该结果可知fb相对于pb有20~30%的性能提升,且使用函数模板后对pb几乎没影响。
   协议\Req Size | 0B | 32B | 64B | 128B | 256B | 512B | 1024B | 2048B | 4096B | 8192B
   -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
   pb(old) | 45k | 46k | 45k | 45k | 45k | 45k | 42k | 41k | 39k | 33k
   pb(new) | 46k | 46k | 47k | 46k | 45k | 44k | 43k | 41k | 39k | 34k
   fb | 58k | 58k | 58k | 58k | 57k | 57k | 55k | 52k | 48k | 43k
   fb相对pb提升百分比 | 26.1% | 26.1% | 23.4% | 26.1% | 26.7% | 29.5% | 27.9% | 26.8% | 23.1% | 26.5%
   
   下表所示为thread_num=64,即并发bthread数目为64的测试结果,可以看到即使message大小为8KB,fb的提升仍有20.4%。
   协议\Req Size | 0B | 32B | 64B | 128B | 256B | 512B | 1024B | 2048B | 4096B | 8192B
   -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
   pb(old) | 526k | 522k | 519k | 511k | 492k | 471k | 441k | 397k | 348k | 282k
   pb(new) | 531k | 521k | 519k | 515k | 490k | 470k | 440k | 399k | 340k | 279k
   fb | 649k | 652k | 650k | 641k | 615k | 586k | 550k | 498k | 416k | 336k
   fb相对pb提升百分比 | 22.2% | 27.3% | 25.2% | 24.5% | 25.5% | 24.7% | 25% | 24.8% | 22.4% | 20.4%
   
   
   另外测试发现即使携带attachment,仍能保持相近的提升百分比,因为attachment不会被序列化。
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org