You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/11/22 13:16:49 UTC
[GitHub] [incubator-doris] spaces-X opened a new issue #7179: [Bug] insert load should have the same behavior with stream load
spaces-X opened a new issue #7179:
URL: https://github.com/apache/incubator-doris/issues/7179
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
version 0.14 or older
### What's Wrong?
The config `default_max_filter_ratio` does not work in the INSERT statement.
If not set `ENABLE_INSERT_STRICT = true`, most of the data may be filtered without notice, which does not match the expectation of `default_max_filter_ratio`
### What You Expected?
Insert load should have the same behavior with stream load.
If `default_max_filter_ratio` is set, the number of filtered rows should not be more than `default_max_filter_ratio` * `total_rows` in the insert load.
### How to Reproduce?
DDL:
1.
```
CREATE TABLE `unique_null_bitmap` (
`dt` int(11) NULL COMMENT "日期分区",
`city_id_bitmap` bitmap BITMAP_UNION COMMENT "city_id_bitmap"
) ENGINE=OLAP
AGGREGATE KEY(`dt`)
COMMENT "交易轻度聚合表"
PARTITION BY RANGE(`dt`)
(PARTITION p20211012 VALUES [("20211011"), ("20211012")),
PARTITION p20211109 VALUES [("20211108"), ("20211109")))
DISTRIBUTED BY HASH(`dt`) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);
```
2.
```
CREATE TABLE `agg_bug_test` (
`dt` bigint(20) NULL COMMENT "",
`id1` bigint(20) NULL COMMENT "",
`id2` bigint(20) NULL COMMENT "",
`id3` bigint(20) NULL COMMENT "",
`first_time` varchar(20) MIN NULL COMMENT "",
`last_time` varchar(20) MAX NULL COMMENT ""
) ENGINE=OLAP
AGGREGATE KEY(`dt`, `id1`, `id2`, `id3`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`id1`, `id2`, `id3`) BUCKETS 10
PROPERTIES (
"replication_num" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);
```
```
insert into agg_bug_test values (20211108,10,10,10,null,null);
insert into agg_bug_test values (20210111,10,10,10,null,null);
```
Insert SQL: 1 row inserted and 8 rows filtered without notice.
Insert transaction should be aborted when `default_max_filter_ratio` is set.
```
insert into unique_null_bitmap (dt,city_id_bitmap) (
(select dt, bitmap_empty() as city_id_bitmap from agg_bug_test where dt = 20211108 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt = 20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt = 20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt = 20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt = 20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt = 20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt = 20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt = 20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt = 20211011 limit 1)
);
```
Query: only returned one row with dt = 20211108
```
select * from unique_null_bitmap
```
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] spaces-X closed issue #7179: [Bug] insert load should have the same behavior with stream load
Posted by GitBox <gi...@apache.org>.
spaces-X closed issue #7179:
URL: https://github.com/apache/incubator-doris/issues/7179
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org