You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "countang (via GitHub)" <gi...@apache.org> on 2025/09/03 10:24:50 UTC

[I] [Feature] Support for Additional Serialization Protocols (e.g. Protobuf) to Improve Proxy/Broker Performance [rocketmq]

countang opened a new issue, #9662:
URL: https://github.com/apache/rocketmq/issues/9662

   ### Is Your Feature Request Related to a Problem?
   
   Currently, RocketMQ uses JSON as the primary serialization format for communication between components such as the Proxy and Broker, especially in newer architectures like the stateless Proxy introduced in RocketMQ 5.x. While JSON is human-readable and widely supported, it has notable performance overhead in terms of both serialization/deserialization latency and CPU usage.
   In high-throughput or low-latency scenarios, the cost of parsing large JSON payloads can become a bottleneck, particularly in the Proxy layer where message routing, metadata handling, and protocol translation happen frequently.
   
   ### Describe the Solution You'd Like
   
   We propose adding support for more efficient serialization protocols, such as Protocol Buffers (Protobuf), Apache Avro, or FlatBuffers, to reduce serialization overhead and improve overall system performance.
   Specifically, we suggest:
   Introduce a pluggable serialization framework in the Proxy and Broker modules, allowing multiple serialization formats (e.g., JSON, Protobuf) to coexist.
   Support Protobuf as an optional or default protocol for internal communication (e.g., between Proxy and Broker, or in client-Proxy interactions).
   Provide configuration options to enable/disable or prioritize specific serialization formats per endpoint or deployment.
   Maintain backward compatibility by supporting JSON as a fallback.
   
   ### Describe Alternatives You've Considered
   
   We have evaluated several alternative approaches to address the serialization performance bottleneck:
   - Optimizing JSON usage (e.g., using faster JSON parsers like Jackson with annotations, or Jsoniter)
   While this can provide moderate performance improvements (e.g., 10–20% faster parsing), it does not fundamentally solve the inherent inefficiencies of text-based formats, such as larger payload size, higher GC pressure, and lack of schema enforcement. It is also limited by JSON's dynamic nature and reflection overhead.
   - Custom binary format with manual serialization
   Designing a lightweight binary protocol tailored to RocketMQ’s internal messages could achieve maximum efficiency. However, this approach sacrifices interoperability, increases maintenance cost, and makes debugging and evolution more difficult. It also lacks tooling support (e.g., schema evolution, code generation).
   Using Apache Avro or FlatBuffers
   - Avro provides schema-based serialization and good performance, but requires a schema registry for efficient operation, adding operational complexity.
   FlatBuffers enables zero-copy parsing and excellent read performance, but has limited language support and higher complexity in message construction.
   While both are viable, they are less mature in the Java ecosystem compared to Protobuf and have smaller community adoption in messaging systems.
   - Sticking with JSON but compressing payloads (e.g., GZIP)
   Compression can reduce network bandwidth usage, but it adds CPU overhead and does not eliminate the cost of parsing — the deserialization still requires full decompression and parsing into objects.
   
   ### Additional Context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Support for Additional Serialization Protocols (e.g. Protobuf) to Improve Proxy/Broker Performance [rocketmq]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9662:
URL: https://github.com/apache/rocketmq/issues/9662#issuecomment-5563311374

   This issue was closed because it has been inactive for 3 days since being marked as stale.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Support for Additional Serialization Protocols (e.g. Protobuf) to Improve Proxy/Broker Performance [rocketmq]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9662:
URL: https://github.com/apache/rocketmq/issues/9662#issuecomment-5533821489

   This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Support for Additional Serialization Protocols (e.g. Protobuf) to Improve Proxy/Broker Performance [rocketmq]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9662: [Feature]  Support for Additional Serialization Protocols (e.g. Protobuf) to Improve Proxy/Broker Performance
URL: https://github.com/apache/rocketmq/issues/9662


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org