You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/12/28 02:51:29 UTC
[GitHub] [doris] yuxuan-luo opened a new issue, #15421: [Bug] JSONB 类型没有对导入的数据进行校验
yuxuan-luo opened a new issue, #15421:
URL: https://github.com/apache/doris/issues/15421
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
1.2.1
### What's Wrong?
使用 stream load 和 insert into 导入数据，类型是 JSONB 。
数据写入时进行JSON格式校验失败，会直接写入 null
### What You Expected?
可以正确进行 JSON格式校验
### How to Reproduce?
CREATE TABLE `test_jsonb` (
`id` int(11) NULL,
`j` jsonb NULL
) ENGINE=OLAP
DUPLICATE KEY(`id`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2",
"disable_auto_compaction" = "false"
);
csv:
1 \N
2 null
3 true
4 false
5 100
6 10000
7 1000000000
8 1152921504606846976
9 6.18
10 "abcd"
11 {}
12 {"k1":"v31", "k2": 300}
13 []
14 [123, 456]
15 ["abc", "def"]
16 [null, true, false, 100, 6.18, "abc"]
17 [{"k1":"v41", "k2": 400}, 1, "a", 3.14]
18 {"k1":"v31", "k2": 300, "a1": [{"k1":"v41", "k2": 400}, 1, "a", 3.14]}
19 ''
20 'abc'
21 abc
22 100x
23 6.a8
24 {x
25 [123, abc]
curl --location-trusted -u admin -H 'max_filter_ratio: 0.3' -T test_jsonb.csv http://127.0.0.1:8030/api/hugo/test_jsonb/_stream_load
{
"TxnId": 2049,
"Label": "7be6f074-6985-4713-be65-cc299107f9d1",
"TwoPhaseCommit": "false",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 25,
"NumberLoadedRows": 25,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 380,
"LoadTimeMs": 24,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 1,
"ReadDataTimeMs": 0,
"WriteDataTimeMs": 6,
"CommitAndPublishTimeMs": 15
}
INSERT INTO test_jsonb VALUES(27, '[123,abc]');
Query OK, 0 rows affected, 1 warning (0.03 sec)
插入显示插入成功，但是在数据表里没有实际插入
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] LemonLiTree commented on issue #15421: [Bug] JSONB 类型没有对导入的数据进行校验
Posted by GitBox <gi...@apache.org>.
LemonLiTree commented on issue #15421:
URL: https://github.com/apache/doris/issues/15421#issuecomment-1367099492
这是跟导入严格模式有关系的：
1. 通过stream load的导入非json数据的情况，strict_mode默认是关闭，这样的话错误数据会转化为null，也就是你上面的那种情况，如果需要filter则可以在stream load 参数中声明：-H "strict_mode: true"。具体可以参考官网的说明：https://doris.apache.org/zh-CN/docs/dev/data-operate/import/import-way/stream-load-manual
2. 通过insert into插入非法数据，你可以看下 show variables like '%enable_insert_strict%';如果是false则返回ok，但插入并不成功，可以SET enable_insert_strict = true;插入则会提示失败。具体可以参考官网的说明：https://doris.apache.org/zh-CN/docs/dev/data-operate/import/import-scenes/load-strict-mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yuxuan-luo closed issue #15421: [Bug] JSONB 类型没有对导入的数据进行校验
Posted by GitBox <gi...@apache.org>.
yuxuan-luo closed issue #15421: [Bug] JSONB 类型没有对导入的数据进行校验
URL: https://github.com/apache/doris/issues/15421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yuxuan-luo commented on issue #15421: [Bug] JSONB 类型没有对导入的数据进行校验
Posted by GitBox <gi...@apache.org>.
yuxuan-luo commented on issue #15421:
URL: https://github.com/apache/doris/issues/15421#issuecomment-1367123814
已解决
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org