You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "Zhengcy05 (via GitHub)" <gi...@apache.org> on 2026/07/13 02:54:04 UTC
[PR] feat: support Firebird BLOB seek in proxy [shardingsphere]
Zhengcy05 opened a new pull request, #39081:
URL: https://github.com/apache/shardingsphere/pull/39081
Special note: Since there are many design changes this time, I sincerely hope someone will carefully review this PR. Thx😘
Fixes #39051.
Changes proposed in this pull request:
-
- Track BLOB read state per connection and per blob handle.
- Make `OPEN_BLOB` register content and cursor state.
- Make `SEEK_BLOB` return the resulting cursor position.
- Make `GET_SEGMENT` read from the current cursor and return EOF correctly.
- Make `INFO_BLOB` read the current BLOB length from the same state.
- Clean up BLOB state on close, cancel, and connection release.
---
## Validation:
- `./mvnw -pl database/protocol/dialect/firebird -am -Dspotless.skip=true -Dtest=FirebirdBlobRegistryTest,FirebirdBlobInfoReturnPacketTest -Dsurefire.failIfNoSpecifiedTests=false test`
- `./mvnw -pl proxy/frontend/dialect/firebird -am -DskipITs -Dspotless.skip=true -Djacoco.skip=true -Dtest=FirebirdOpenBlobCommandExecutorTest,FirebirdCloseBlobCommandExecutorTest,FirebirdCancelBlobCommandExecutorTest,FirebirdGetBlobSegmentCommandExecutorTest,FirebirdSeekBlobCommandExecutorTest,FirebirdBlobInfoExecutorTest,FirebirdFrontendEngineTest,FirebirdAuthenticationEngineTest,FirebirdCommandExecuteEngineTest -Dsurefire.failIfNoSpecifiedTests=false -DargLine='-javaagent:/Users/didi/.m2/repository/net/bytebuddy/byte-buddy-agent/1.18.10/byte-buddy-agent-1.18.10.jar -Djdk.attach.allowAttachSelf=true -XX:+EnableDynamicAgentLoading' test`
- `./mvnw spotless:apply -Pcheck -T1C`
- `./mvnw checkstyle:check -Pcheck -T1C`
```
Before committing this PR, I'm sure that I have checked the following options:
- [x] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
- [x] I have self-reviewed the commit code.
- [ ] I have (or in comment I request) added corresponding labels for the pull request.
- [ ] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
- [ ] I have made corresponding changes to the documentation.
- [x] I have added corresponding unit tests for my changes.
- [ ] I have updated the Release Notes of the current development version. For more details, see [Update Release Note](https://shardingsphere.apache.org/community/en/involved/contribute/contributor/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support Firebird BLOB seek in proxy [shardingsphere]
Posted by "terrymanu (via GitHub)" <gi...@apache.org>.
terrymanu commented on PR #39081:
URL: https://github.com/apache/shardingsphere/pull/39081#issuecomment-5100735795
Thanks for the work on this PR. #39050 has now been merged, but it does not implement op_seek_blob; the current master still explicitly reports SEEK BLOB as unsupported, so #39051 should remain open.
However, #39050 replaced the BLOB state-management and package structure on which this PR is based, and most of the broader BLOB fixes included here are now already covered. Updating this branch would effectively require a new implementation rather than a normal rebase.
I suggest closing this PR as superseded by the BLOB architecture introduced in #39050—not as fixed or duplicate—and opening a focused follow-up PR against the latest master. The new PR should extend the current Proxy-side BLOB read state with cursor and BLOB-type information, implement the three seek modes, preserve deferred-handle resolution, and reject seek for non-stream BLOBs.
If you would still like to work on this, we would be happy for you to continue with that focused follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support Firebird BLOB seek in proxy [shardingsphere]
Posted by "makssent (via GitHub)" <gi...@apache.org>.
makssent commented on PR #39081:
URL: https://github.com/apache/shardingsphere/pull/39081#issuecomment-4956605653
I've also been working on BLOB support in the Firebird proxy (#39050). The original issue only covered `op_seek_blob`, but since this PR implements the full BLOB flow, I'd like to share two bugs I ran into while testing BLOB scenarios - both reproduce on this branch as well:
**1. The deferred BLOB handle `0xFFFF` (INVALID_OBJECT) is not resolved.**
Jaybird 5.0.7+ pipelines BLOB operations: it sends `op_put_segment` / `op_info_blob` / `op_get_segment` right after `op_create_blob2` / `op_open_blob2` without waiting for the response, using the placeholder `0xFFFF` (`INVALID_OBJECT`) in place of the not-yet-known handle. The Firebird server substitutes the handle of the most recently created object in this case, while the proxy looks up `0xFFFF` literally:
- on write, `op_put_segment` finds no blob and silently drops the segment → the row is inserted with **empty BLOBs**;
- on read, `op_get_segment` returns empty data and never reports the end-of-blob state (`p_resp_object = 2`), so the client's read loop **never terminates**.
Reproducible with Jaybird 5.0.10: INSERT stores empty BLOBs, and a subsequent SELECT of a BLOB column hangs.
**2. The `op_get_segment` response declares the alignment padding as part of the data.**
The response data must be `[2-byte little-endian length][segment bytes]`, with the 4-byte alignment padding written **outside** the declared length. Currently the declared length includes the padding, so Jaybird reads the padding as data and runs past the buffer on any segment whose length % 4 == 1 or 3:
```
java.lang.ArrayIndexOutOfBoundsException: Index 26 out of bounds for length 26
at org.firebirdsql.gds.VaxEncoding.iscVaxInteger2(VaxEncoding.java:165)
at org.firebirdsql.gds.ng.wire.version10.V10InputBlob.getSegment(V10InputBlob.java:136)
at org.firebirdsql.jdbc.field.FBBlobField.getBytesInternal(FBBlobField.java:149)
```
Reproducible with Jaybird 5.0.6 by reading back a 21-byte BLOB (2 + 21 = 23 bytes of payload, 3 bytes of padding, declared length 26). Segments whose length % 4 == 0 or 2 survive by luck — that's why small ASCII test payloads often pass. Large BLOBs always fail: the client requests chunks of 32765 bytes (32765 % 4 == 1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support Firebird BLOB seek in proxy [shardingsphere]
Posted by "Zhengcy05 (via GitHub)" <gi...@apache.org>.
Zhengcy05 commented on PR #39081:
URL: https://github.com/apache/shardingsphere/pull/39081#issuecomment-4976171969
Thank you very much for this detailed report — both bugs have been reproduced on my end and have now been fixed.
**1. Unresolved delay handle '0xFFFF'**
Root cause: Both 'FirebirdCreateBlobCommandExecutor' and 'FirebirdOpenBlobCommandExecutor' generate a new handle and return immediately, but neither records the handle for subsequent operations. When Jaybird uses '0xFFFF' to pipeline before the response returns, all subsequent executors (put-segment, get-segment, seek, close, cancel) directly look for '0xFFFF' and get an empty value—causing silent drops (data) during writing and deadlocks when reading.
Fix: 'FirebirdBlobRegistry' now tracks the last created/opened blob handle by connection. 'create_blob2' and 'open_blob2' are both recorded using 'setLastBlobHandle()'. All five subsequent executors now call 'resolveBlobHandle()' before each registry/cache lookup, and when the client sends '0xFFFF', it returns the last handle—which matches how a real Firebird server replaces the recently created object handle.
**2. Incorrect fill length in the 'op_get_segment' response**
Root cause: 'FirebirdGetBlobSegmentResponsePacket.write()' adds its own internal 4-byte alignment padding after a 2-byte prefix. Because 'FirebirdGenericResponsePacket' encapsulates response data with its XDR length prefix, the internal padding is included in that length. For a 21-byte fragment: declared length = 2 + 21 + 3 (padded) = 26, so Jaybird reads the 21-byte data and interprets the 3 padding bytes as the prefix -> for the next 2-byte fragment length, causing buffer overrun.
Fix: Removed internal padding from 'FirebirdGetBlobSegmentResponsePacket.write()'. The generic response encapsulation has applied the correct external alignment via '(4 - length) & 3', which lies outside the declared data length, which is exactly what the protocol requires.
Now, both fixes include dedicated unit tests, such as the 'assertExecuteWithDeferredHandle' test, which verifies '0xFFFF' parsing on all five subsequent executors, as well as updated response packet tests for misaligned fragments. If you wish to coordinate BLOB testing between these two PRs, I am happy to help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support Firebird BLOB seek in proxy [shardingsphere]
Posted by "Zhengcy05 (via GitHub)" <gi...@apache.org>.
Zhengcy05 commented on PR #39081:
URL: https://github.com/apache/shardingsphere/pull/39081#issuecomment-4964774783
> I've also been working on BLOB support in the Firebird proxy (#39050). The original issue only covered `op_seek_blob`, but since this PR implements the full BLOB flow, I'd like to share two bugs I ran into while testing BLOB scenarios - both reproduce on this branch as well:我还在做 Firebird 代理中的 BLOB 支持 #39050 （ ）。最初的问题只涵盖`了 op_seek_blob`，但由于这个 PR 实现了完整的 BLOB 流程，我想分享我在测试 BLOB 场景时遇到的两个 bug——这两个错误也在本分支中重现：
>
> **1. The deferred BLOB handle `0xFFFF` (INVALID_OBJECT) is not resolved.1. 延迟的 BLOB 句柄 `0xFFFF`（INVALID_OBJECT）未被解决。**
>
> Jaybird 5.0.7+ pipelines BLOB operations: it sends `op_put_segment` / `op_info_blob` / `op_get_segment` right after `op_create_blob2` / `op_open_blob2` without waiting for the response, using the placeholder `0xFFFF` (`INVALID_OBJECT`) in place of the not-yet-known handle. The Firebird server substitutes the handle of the most recently created object in this case, while the proxy looks up `0xFFFF` literally:Jaybird 5.0.7+ 流水线 BLOB 操作：它在 `op_create_blob2` / `op_open_blob2` 之后立即发送 `op_put_segment` / `op_info_blob` / `op_get_segment`，无需等待响应，使用占位符 `0xFFFF` （`INVALID_OBJECT`） 代替尚未出现的句柄。在这种情况下，Firebird 服务器会用最近创建对象的句柄替换，而代理则直接查找 `0xFFFF`：
>
> * on write, `op_put_segment` finds no blob and silently drops the segment → the row is inserted with **empty BLOBs**;写入时，`op_put_segment` 找不到 blob，→插入**行**时，会悄无声息地丢弃该段;
> * on read, `op_get_segment` returns empty data and never reports the end-of-blob state (`p_resp_object = 2`), so the client's read loop **never terminates**.读取时，`op_get_segment` 返回空数据，且从不报告 blob 结束状态（`p_resp_object = 2`），因此客户端的读取循环**永远不会终止 **。
>
> Reproducible with Jaybird 5.0.10: INSERT stores empty BLOBs, and a subsequent SELECT of a BLOB column hangs.可用 Jaybird 5.0.10 重现：INSERT 存储空的 BLOB，随后 BLOB 列的 SELECT 会被挂起。
>
> **2. The `op_get_segment` response declares the alignment padding as part of the data.2. `op_get_segment` 响应声明对齐填充作为数据的一部分。**
>
> The response data must be `[2-byte little-endian length][segment bytes]`, with the 4-byte alignment padding written **outside** the declared length. Currently the declared length includes the padding, so Jaybird reads the padding as data and runs past the buffer on any segment whose length % 4 == 1 or 3:响应数据必须是 `[2 字节小端长度][段字节]，`4 字节对齐填充写**入声明长度之外 **。目前声明的长度包含填充，因此 Jaybird 将填充读取为数据，并对长度 % 4 == 1 或 3 的段运行缓冲区：
>
> ```
> java.lang.ArrayIndexOutOfBoundsException: Index 26 out of bounds for length 26
> at org.firebirdsql.gds.VaxEncoding.iscVaxInteger2(VaxEncoding.java:165)
> at org.firebirdsql.gds.ng.wire.version10.V10InputBlob.getSegment(V10InputBlob.java:136)
> at org.firebirdsql.jdbc.field.FBBlobField.getBytesInternal(FBBlobField.java:149)
> ```
>
> Reproducible with Jaybird 5.0.6 by reading back a 21-byte BLOB (2 + 21 = 23 bytes of payload, 3 bytes of padding, declared length 26). Segments whose length % 4 == 0 or 2 survive by luck — that's why small ASCII test payloads often pass. Large BLOBs always fail: the client requests chunks of 32765 bytes (32765 % 4 == 1).通过读取一个 21 字节的 BLOB（2 + 21 = 23 字节有效载荷，3 字节填充，声明长度 26）可通过 Jaybird 5.0.6 复现。长度为 % 4 == 0 或 2 的段靠运气存活——这也是为什么小型 ASCII 测试载荷经常能通过。大型 BLOB 总是失败：客户端请求 32765 字节的块（32765 % 4 == 1）。
Thank you very much, I will make corrections
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support Firebird BLOB seek in proxy [shardingsphere]
Posted by "terrymanu (via GitHub)" <gi...@apache.org>.
terrymanu closed pull request #39081: feat: support Firebird BLOB seek in proxy
URL: https://github.com/apache/shardingsphere/pull/39081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org