You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "azexcy (via GitHub)" <gi...@apache.org> on 2023/02/07 13:02:13 UTC
[GitHub] [shardingsphere] azexcy opened a new pull request, #24049: Fix unicode cannot be decoded correctly at TestDecodingPlugin
azexcy opened a new pull request, #24049:
URL: https://github.com/apache/shardingsphere/pull/24049
Changes proposed in this pull request:
- Fix unicode cannot be decoded correctly at TestDecodingPlugin
- E2E add unicode case
---
Before committing this PR, I'm sure that I have checked the following options:
- [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
- [ ] I have self-reviewed the commit code.
- [ ] I have (or in comment I request) added corresponding labels for the pull request.
- [ ] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
- [ ] I have made corresponding changes to the documentation.
- [ ] I have added corresponding unit tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #24049: Fix unicode cannot be decoded correctly at TestDecodingPlugin
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on code in PR #24049:
URL: https://github.com/apache/shardingsphere/pull/24049#discussion_r1101416502
##########
kernel/data-pipeline/dialect/postgresql/src/main/java/org/apache/shardingsphere/data/pipeline/postgresql/ingest/wal/decode/TestDecodingPlugin.java:
##########
@@ -208,26 +211,60 @@ private String readNextSegment(final ByteBuffer data) {
return eventType.toString();
}
+ private String readNextJson(final ByteBuffer data) {
Review Comment:
Added in unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz merged pull request #24049: Fix unicode char and special char decoding in TestDecodingPlugin for PostgreSQL
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz merged PR #24049:
URL: https://github.com/apache/shardingsphere/pull/24049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #24049: Fix unicode cannot be decoded correctly at TestDecodingPlugin
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #24049:
URL: https://github.com/apache/shardingsphere/pull/24049#discussion_r1099655201
##########
kernel/data-pipeline/dialect/postgresql/src/main/java/org/apache/shardingsphere/data/pipeline/postgresql/ingest/wal/decode/TestDecodingPlugin.java:
##########
@@ -208,26 +211,60 @@ private String readNextSegment(final ByteBuffer data) {
return eventType.toString();
}
+ private String readNextJson(final ByteBuffer data) {
+ data.get();
+ int offset = 0;
+ int startPosition = data.position();
+ while (data.hasRemaining()) {
+ char c = (char) data.get();
+ if ('}' != c) {
Review Comment:
`c`, `c2` and `c3` could be removed and embedded in `if` condition
##########
kernel/data-pipeline/dialect/postgresql/src/main/java/org/apache/shardingsphere/data/pipeline/postgresql/ingest/wal/decode/TestDecodingPlugin.java:
##########
@@ -208,26 +211,60 @@ private String readNextSegment(final ByteBuffer data) {
return eventType.toString();
}
+ private String readNextJson(final ByteBuffer data) {
+ data.get();
+ int offset = 0;
+ int startPosition = data.position();
+ while (data.hasRemaining()) {
+ char c = (char) data.get();
+ if ('}' != c) {
+ offset++;
+ continue;
+ }
+ char c2 = (char) data.get();
+ if ('\'' != c2) {
+ offset++;
+ continue;
+ }
+ char c3 = (char) data.get();
+ if (' ' == c3) {
Review Comment:
If json text contains `' '` (blank char), then does the parsing work?
##########
kernel/data-pipeline/dialect/postgresql/src/main/java/org/apache/shardingsphere/data/pipeline/postgresql/ingest/wal/decode/TestDecodingPlugin.java:
##########
@@ -208,26 +211,60 @@ private String readNextSegment(final ByteBuffer data) {
return eventType.toString();
}
+ private String readNextJson(final ByteBuffer data) {
Review Comment:
It's better to set more complicated json text in E2E
##########
kernel/data-pipeline/dialect/postgresql/src/main/java/org/apache/shardingsphere/data/pipeline/postgresql/ingest/wal/decode/TestDecodingPlugin.java:
##########
@@ -208,26 +211,60 @@ private String readNextSegment(final ByteBuffer data) {
return eventType.toString();
}
+ private String readNextJson(final ByteBuffer data) {
+ data.get();
+ int offset = 0;
+ int startPosition = data.position();
+ while (data.hasRemaining()) {
+ char c = (char) data.get();
+ if ('}' != c) {
+ offset++;
+ continue;
+ }
+ char c2 = (char) data.get();
+ if ('\'' != c2) {
+ offset++;
+ continue;
+ }
+ char c3 = (char) data.get();
+ if (' ' == c3) {
+ offset++;
+ return getStringSegment(data, startPosition, offset).replace("''", "'");
+ }
+ offset++;
+ }
+ return null;
+ }
+
+ private String getStringSegment(final ByteBuffer data, final int startPosition, final int offset) {
Review Comment:
`getStringSegment` could be `readStringSegment`, keep consistent with other methods name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter commented on pull request #24049: Fix unicode cannot be decoded correctly at TestDecodingPlugin
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24049:
URL: https://github.com/apache/shardingsphere/pull/24049#issuecomment-1421934981
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#24049](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (db2ef50) into [master](https://codecov.io/gh/apache/shardingsphere/commit/db0b6f356d8cbff80ec8cd37165c14d471397e77?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (db0b6f3) will **decrease** coverage by `0.01%`.
> The diff coverage is `16.17%`.
```diff
@@ Coverage Diff @@
## master #24049 +/- ##
============================================
- Coverage 50.07% 50.07% -0.01%
+ Complexity 1564 1562 -2
============================================
Files 3245 3242 -3
Lines 53221 53240 +19
Branches 9800 9803 +3
============================================
+ Hits 26653 26658 +5
- Misses 24225 24241 +16
+ Partials 2343 2341 -2
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...crypt/metadata/EncryptSchemaMetaDataDecorator.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvZW5jcnlwdC9jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9lbmNyeXB0L21ldGFkYXRhL0VuY3J5cHRTY2hlbWFNZXRhRGF0YURlY29yYXRvci5qYXZh) | `100.00% <ø> (ø)` | |
| [...ding/metadata/ShardingSchemaMetaDataDecorator.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvc2hhcmRpbmcvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2hhcmRpbmcvbWV0YWRhdGEvU2hhcmRpbmdTY2hlbWFNZXRhRGF0YURlY29yYXRvci5qYXZh) | `75.86% <ø> (+6.89%)` | :arrow_up: |
| [...ng/metadata/reviser/ShardingConstraintReviser.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvc2hhcmRpbmcvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2hhcmRpbmcvbWV0YWRhdGEvcmV2aXNlci9TaGFyZGluZ0NvbnN0cmFpbnRSZXZpc2VyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...harding/metadata/reviser/ShardingIndexReviser.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvc2hhcmRpbmcvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2hhcmRpbmcvbWV0YWRhdGEvcmV2aXNlci9TaGFyZGluZ0luZGV4UmV2aXNlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...tor/reviser/constraint/ConstraintReviseEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9kYXRhYmFzZS9zY2hlbWEvZGVjb3JhdG9yL3JldmlzZXIvY29uc3RyYWludC9Db25zdHJhaW50UmV2aXNlRW5naW5lLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ema/decorator/reviser/index/IndexReviseEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9kYXRhYmFzZS9zY2hlbWEvZGVjb3JhdG9yL3JldmlzZXIvaW5kZXgvSW5kZXhSZXZpc2VFbmdpbmUuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...rator/reviser/table/TableMetaDataReviseEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9kYXRhYmFzZS9zY2hlbWEvZGVjb3JhdG9yL3JldmlzZXIvdGFibGUvVGFibGVNZXRhRGF0YVJldmlzZUVuZ2luZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...single/metadata/SingleSchemaMetaDataDecorator.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL3NpbmdsZS9jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9zaW5nbGUvbWV0YWRhdGEvU2luZ2xlU2NoZW1hTWV0YURhdGFEZWNvcmF0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ngle/metadata/reviser/SingleConstraintReviser.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL3NpbmdsZS9jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9zaW5nbGUvbWV0YWRhdGEvcmV2aXNlci9TaW5nbGVDb25zdHJhaW50UmV2aXNlci5qYXZh) | `0.00% <0.00%> (-50.00%)` | :arrow_down: |
| [...stgresql/ingest/wal/decode/TestDecodingPlugin.java](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvZGlhbGVjdC9wb3N0Z3Jlc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kYXRhL3BpcGVsaW5lL3Bvc3RncmVzcWwvaW5nZXN0L3dhbC9kZWNvZGUvVGVzdERlY29kaW5nUGx1Z2luLmphdmE=) | `64.82% <26.66%> (-9.34%)` | :arrow_down: |
| ... and [15 more](https://codecov.io/gh/apache/shardingsphere/pull/24049?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on pull request #24049: Fix unicode cannot be decoded correctly at TestDecodingPlugin
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on PR #24049:
URL: https://github.com/apache/shardingsphere/pull/24049#issuecomment-1420746408
The single quote need be escape if they exists in the content, just like.
```
UPDATE test.t_order
SET status = '''中文'''
WHERE order_id = xxx;
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #24049: Fix unicode cannot be decoded correctly at TestDecodingPlugin
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on code in PR #24049:
URL: https://github.com/apache/shardingsphere/pull/24049#discussion_r1101416212
##########
kernel/data-pipeline/dialect/postgresql/src/main/java/org/apache/shardingsphere/data/pipeline/postgresql/ingest/wal/decode/TestDecodingPlugin.java:
##########
@@ -208,26 +211,60 @@ private String readNextSegment(final ByteBuffer data) {
return eventType.toString();
}
+ private String readNextJson(final ByteBuffer data) {
+ data.get();
+ int offset = 0;
+ int startPosition = data.position();
+ while (data.hasRemaining()) {
+ char c = (char) data.get();
+ if ('}' != c) {
+ offset++;
+ continue;
+ }
+ char c2 = (char) data.get();
+ if ('\'' != c2) {
+ offset++;
+ continue;
+ }
+ char c3 = (char) data.get();
+ if (' ' == c3) {
Review Comment:
I improved json parse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #24049: Fix unicode cannot be decoded correctly at TestDecodingPlugin
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on code in PR #24049:
URL: https://github.com/apache/shardingsphere/pull/24049#discussion_r1101416502
##########
kernel/data-pipeline/dialect/postgresql/src/main/java/org/apache/shardingsphere/data/pipeline/postgresql/ingest/wal/decode/TestDecodingPlugin.java:
##########
@@ -208,26 +211,60 @@ private String readNextSegment(final ByteBuffer data) {
return eventType.toString();
}
+ private String readNextJson(final ByteBuffer data) {
Review Comment:
Added in unit test and E2E
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org