You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "gaoran10 (via GitHub)" <gi...@apache.org> on 2025/09/29 03:05:53 UTC

[PR] [improve][PIP] Update the schema ID format [pulsar]

gaoran10 opened a new pull request, #24798:
URL: https://github.com/apache/pulsar/pull/24798

   ### Motivation
   
   Currently, the Schema ID has two types, and they have different schema ID formats:
   - value schema ID (external schema ID data)
   - key-value schema ID (keySchemaId length (4 bytes) + keySchemaId data + valueSchemaId length (4 bytes) + valueSchemaId data)
   
   This will cause a problem; if users want to use the schema ID in message metadata, they can't distinguish the value schema ID from the key-value schema ID.
   
   We can add a magic header for the encoding format.
   
   ### Modifications
   
   Unify the value schema ID and key-value schema ID in this format:
   magic_byte(1 byte) + keySchemaIdLength(4 bytes) + keySchemaId + valueSchemaIdLength(4 bytes) + valueSchemaId
   
   The first version magic byte is 0x0.
   
   ### Does this pull request potentially affect one of the following parts:
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [x] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][client] PIP-420: Update the schema ID format [pulsar]

Posted by "gaoran10 (via GitHub)" <gi...@apache.org>.
gaoran10 merged PR #24798:
URL: https://github.com/apache/pulsar/pull/24798


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][client] PIP-420: Update the schema ID format [pulsar]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24798:
URL: https://github.com/apache/pulsar/pull/24798#issuecomment-3383870785

   ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/24798?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `84.61538%` with `4 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 74.02%. Comparing base ([`e44e084`](https://app.codecov.io/gh/apache/pulsar/commit/e44e084faf66738a22d61892a86d9fc9943bc484?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`bf3d68f`](https://app.codecov.io/gh/apache/pulsar/commit/bf3d68f2701e94312ad1fa9fe8d0f0c0f22fd471?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 13 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/24798?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...main/java/org/apache/pulsar/client/api/Schema.java](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&filepath=pulsar-client-api%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fapi%2FSchema.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9jbGllbnQvYXBpL1NjaGVtYS5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../org/apache/pulsar/common/schema/SchemaIdUtil.java](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&filepath=pulsar-client-api%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcommon%2Fschema%2FSchemaIdUtil.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9jb21tb24vc2NoZW1hL1NjaGVtYUlkVXRpbC5qYXZh) | 93.75% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ava/org/apache/pulsar/client/impl/MessageImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&filepath=pulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FMessageImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL01lc3NhZ2VJbXBsLmphdmE=) | 75.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   [![Impacted file tree graph](https://app.codecov.io/gh/apache/pulsar/pull/24798/graphs/tree.svg?width=650&height=150&src=pr&token=acYqCpsK9J&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #24798       +/-   ##
   =============================================
   + Coverage     38.36%   74.02%   +35.66%     
   - Complexity    13171    33799    +20628     
   =============================================
     Files          1854     1886       +32     
     Lines        144870   148710     +3840     
     Branches      16808    17256      +448     
   =============================================
   + Hits          55574   110082    +54508     
   + Misses        81752    29812    -51940     
   - Partials       7544     8816     +1272     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/24798/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/24798/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.42% <7.69%> (+0.20%)` | :arrow_up: |
   | [systests](https://app.codecov.io/gh/apache/pulsar/pull/24798/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/24798/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.79% <84.61%> (+39.17%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/24798?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...he/pulsar/client/impl/TypedMessageBuilderImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&filepath=pulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FTypedMessageBuilderImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1R5cGVkTWVzc2FnZUJ1aWxkZXJJbXBsLmphdmE=) | `86.80% <100.00%> (+25.53%)` | :arrow_up: |
   | [.../org/apache/pulsar/common/schema/SchemaIdUtil.java](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&filepath=pulsar-client-api%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fcommon%2Fschema%2FSchemaIdUtil.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9jb21tb24vc2NoZW1hL1NjaGVtYUlkVXRpbC5qYXZh) | `93.75% <93.75%> (ø)` | |
   | [...ava/org/apache/pulsar/client/impl/MessageImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&filepath=pulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FMessageImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL01lc3NhZ2VJbXBsLmphdmE=) | `75.43% <75.00%> (+22.05%)` | :arrow_up: |
   | [...main/java/org/apache/pulsar/client/api/Schema.java](https://app.codecov.io/gh/apache/pulsar/pull/24798?src=pr&el=tree&filepath=pulsar-client-api%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fapi%2FSchema.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9jbGllbnQvYXBpL1NjaGVtYS5qYXZh) | `80.00% <0.00%> (+15.00%)` | :arrow_up: |
   
   ... and [1455 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/24798/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org