You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/10/04 03:34:31 UTC
[PR] [cdc] Decode MySQL SET bitmaps wider than 31 bits in canal/aliyun [paimon]
jackylee-ch opened a new pull request, #10372:
URL: https://github.com/apache/paimon/pull/10372
### Purpose
`CanalFieldParser` and `AliyunFieldParser` decode a MySQL `SET` value from its
bitmap representation with `Integer.parseInt(value)`. A MySQL `SET` column may
declare up to **64** members, so selecting a member at bit 31 or higher yields a
bitmap value greater than `Integer.MAX_VALUE`. `Integer.parseInt` then threw
`NumberFormatException`, failing the whole canal / aliyun-DTS sync job for an
otherwise valid row.
This parses the bitmap as a `long` and widens `getSetValuesByIndex` to match. The
decoding loop already operated in long terms (`indexes != 0L`, `indexes % 2L`,
`indexes >>> 1`), and the method's own comment already states the value is
"a bit string conversion from the long" — the parse was the only place still
narrowed to `int`. Values within the int range decode exactly as before.
### Tests
`CanalFieldParserSetTest` / `AliyunFieldParserSetTest` decode a 40-member `SET`:
a high bit (2^32) and a low+high combination both resolve to the correct members,
and a small in-range value is unchanged. The high-bit cases throw
`NumberFormatException` before the change.
### API and Format
No change.
### Documentation
No change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] Decode MySQL SET bitmaps wider than 31 bits in canal/aliyun [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #10372:
URL: https://github.com/apache/paimon/pull/10372#discussion_r4177919163
##########
paimon-flink/paimon-flink-cdc/src/main/java/org/apache/paimon/flink/action/cdc/format/canal/CanalFieldParser.java:
##########
@@ -57,7 +57,7 @@ protected static byte[] convertGeoType2WkbArray(byte[] mysqlGeomBytes) {
protected static String convertSet(String value, String mysqlType) {
// mysql set type value can be filled with more than one, value is a bit string conversion
// from the long
- int indexes = Integer.parseInt(value);
+ long indexes = Long.parseLong(value);
Review Comment:
[P2] Parse the SET bitmap as unsigned so the 64th member is supported. MySQL permits 64 SET members (https://dev.mysql.com/doc/refman/8.4/en/set.html). Canal's RowsLogBuffer uses getUlong64 for 8-byte SET values, and LogBuffer.getUlong64 returns BigInteger; LogEventConvert serializes it using String.valueOf, so selecting member 64 produces "9223372036854775808", not a negative signed-long string. This new Long.parseLong still throws NumberFormatException for that valid event, leaving the claimed wide-SET CDC failure unfixed at the upper boundary. I reproduced it through CanalRecordParser.buildSchema with mysqlType set('c0',...,'c63') and INSERT data s="9223372036854775808". Replacing this call with Long.parseUnsignedLong makes the same event decode to [c63]; the existing >>> shift loop already supports the resulting bits. Please use unsigned parsing and add actual-event tests for bit 63 and all 64 bits. Upstream encoding: https://github.com/alibaba/canal/blob/master/dbsync/src/mai
n/java/com/taobao/tddl/dbsync/binlog/event/RowsLogBuffer.java#L868 and https://github.com/alibaba/canal/blob/master/dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/LogBuffer.java#L996 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] Parse MySQL SET bitmaps as unsigned 64-bit in canal/aliyun [paimon]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #10372:
URL: https://github.com/apache/paimon/pull/10372#issuecomment-5982008121
Good catch. The 64th SET member sets bit 63 (decimal 2^63 > Long.MAX_VALUE), so even the signed-long parse threw NumberFormatException. Switched to `Long.parseUnsignedLong`, so the full 64-member range decodes — the loop already shifts unsigned (`>>>`). Added a `SET(64)` case asserting the 64th member resolves to `[c63]`; it fails on the signed parse.
On scope: the production path is Canal (`CanalRecordParser` → `convertSet`); I've tied the claim there. `AliyunFieldParser` is byte-identical duplicated code, fixed in parallel to stay correct for when it is wired to a record parser — its unit test exercises the helper, not a DTS job. Pushed in 50c03c1af.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] Parse MySQL SET bitmaps as unsigned 64-bit in canal/aliyun [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10372:
URL: https://github.com/apache/paimon/pull/10372#issuecomment-5994697722
CI failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] Parse MySQL SET bitmaps as unsigned 64-bit in canal/aliyun [paimon]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #10372:
URL: https://github.com/apache/paimon/pull/10372#issuecomment-5995316829
Closing this to focus my contributions on a few areas. This is a genuine fix, but CDC isn't where I'm concentrating right now -- happy to reopen it if it's useful to someone. Thanks for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] Parse MySQL SET bitmaps as unsigned 64-bit in canal/aliyun [paimon]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch closed pull request #10372: [cdc] Parse MySQL SET bitmaps as unsigned 64-bit in canal/aliyun
URL: https://github.com/apache/paimon/pull/10372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org