You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@bookkeeper.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/05/07 20:36:27 UTC

[PR] Migrate DataFormats from protobuf-java to LightProto [bookkeeper]

merlimat opened a new pull request, #4779:
URL: https://github.com/apache/bookkeeper/pull/4779

   ## Summary
   
   - Migrate `DataFormats.proto` and `DbLedgerStorageDataFormats.proto` in `bookkeeper-proto` from the Google `protobuf-java` toolchain to [StreamNative LightProto](https://github.com/streamnative/lightproto) (`io.streamnative.lightproto:lightproto-maven-plugin:0.7.0`).
   - Wire protocol (`BookkeeperProtocol.proto`) is intentionally left on protobuf-java for now; this lays the groundwork for migrating it next.
   - Uses LightProto's opt-in `generateTextFormat=true`, so on-disk and ZK-stored proto2 TextFormat payloads (cookies, auditor votes, lock data, underreplication entries, layout) keep their existing format and round-trip byte-identically.
   
   ## What this changes
   
   - `bookkeeper-proto/pom.xml`: protobuf-maven-plugin now compiles only `BookkeeperProtocol.proto`; lightproto-maven-plugin compiles `DataFormats.proto` and `DbLedgerStorageDataFormats.proto`.
   - All Java callers of `org.apache.bookkeeper.proto.DataFormats.*` and `org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageDataFormats.LedgerData` updated to the LightProto API:
     - `Foo.newBuilder()...build()` → `new Foo()...`
     - `parseFrom(bytes)` → `new Foo(); foo.parseFrom(bytes);`
     - Boolean `getX()` accessors → `isX()` (proto2 convention in LightProto)
     - `ByteString` → `byte[]` / `ByteBuf`
     - `TextFormat.printer().print()` / `TextFormat.merge()` → `toTextFormat()` / `parseFromTextFormat()`
     - v3 ledger metadata uses a hand-rolled length-prefixed delimited reader/writer matching `writeDelimitedTo`/`mergeDelimitedFrom`.
   - Minor public API change: `BookKeeper.DigestType.toProtoDigestType` now returns the LightProto-generated enum (same constants, new package `org.apache.bookkeeper.proto.LedgerMetadataFormat.DigestType`).
   
   ## Test plan
   
   - [x] `bookkeeper-proto` builds cleanly with both plugins
   - [x] `mvn apache-rat:check` passes (RAT excludes updated for new generated tree)
   - [x] `bookkeeper-server` and `microbenchmarks` compile
   - [x] `TestLedgerMetadataSerDe` (v1/v2/v3 round-trips, including v2 TextFormat) — 11/11
   - [x] `TestLedgerUnderreplicationManager` + auditor — 29/29
   - [x] `CookieTest` + `DbLedgerStorage*` — 46/46
   - [x] `BookKeeperTest`, `BookieInitializationTest`, `BookieClientTest`, `DigestManagerTest` — 77/77 (2 skipped)
   - [ ] CI green


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Migrate DataFormats from protobuf-java to LightProto [bookkeeper]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.

merlimat merged PR #4779:
URL: https://github.com/apache/bookkeeper/pull/4779


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Migrate DataFormats from protobuf-java to LightProto [bookkeeper]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.

lhotari commented on PR #4779:
URL: https://github.com/apache/bookkeeper/pull/4779#issuecomment-4403769605

   @merlimat I missed `ByteBufUtil.getBytes` usage in this PR. I'd assume that it would make sense to pass `ByteBuf` instances directly to LightProto while taking the ref count handling into account. I added some comments in #4780.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org