You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "tangbinyeer (via GitHub)" <gi...@apache.org> on 2023/12/07 07:29:07 UTC
[I] [Bug] 聚合模型BITMAP类型聚合结果不正确 [doris]
tangbinyeer opened a new issue, #28118:
URL: https://github.com/apache/doris/issues/28118
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
2.0.2
### What's Wrong?
有一张表,主要用于根据指定key分组后记录去重对hash计数,因此作为bitmap类型设定,表大概如下:
```
CREATE TABLE IF NOT EXISTS bsc_tx_pair2(
period DATETIME COMMENT '统计周期起始时间',
type VARCHAR(16) COMMENT '交易类型,本币:tx,内部:inner,ERC20代币:erc20',
from_addr VARCHAR(64) COMMENT '交易发送地址',
to_addr VARCHAR(64) COMMENT '交易接收地址',
token VARCHAR(64) DEFAULT '' COMMENT '代币地址',
pair BITMAP BITMAP_UNION COMMENT 'from,to有序组成的对,用于计算对手数',
tx_count BIGINT SUM COMMENT '交易量',
hashes BITMAP BITMAP_UNION COMMENT '涉及的交易hash'
)
AGGREGATE KEY(period, type, from_addr, to_addr, token)
PARTITION BY RANGE(period) ()
DISTRIBUTED BY HASH(type, token) BUCKETS AUTO
```
使用相同的程序(程序可以控制一次写入多少数据)进行数据写入,
当我单批次插入数据后:
使用如下查询语句进行查询:
```
SELECT `type`,
`token`,
`from_addr`,
`to_addr`,
BITMAP_UNION_COUNT(`pair`),
SUM(`tx_count`),
BITMAP_UNION_COUNT(`hashes`)
FROM bsc_tx_pair
where from_addr='0x6488aa4d1955ee33403f8ccb1d4de5fb97c7ade2' and to_addr='0x0000000000000000000000000000000000001001'
GROUP BY `type`, `token`, `from_addr`, `to_addr`;
```
其结果是我想要得,也是正确得(SUM(tx_count) <= BITMAP_UNION_COUNT(hashes))

但是当我分批多次插入数据后(写入了另一个相同的表,数据也是一样,只是分批写入),查询结果是错误的,
明显的错误就是,这里的 tx_count不应该小于hashes count

### What You Expected?
请问是bitmap没有用对?
还是聚合模型在合并数据时对bitmap的处理存在问题,导致数据不准?
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 聚合模型BITMAP类型聚合结果不正确 [doris]
Posted by "tangbinyeer (via GitHub)" <gi...@apache.org>.
tangbinyeer closed issue #28118: [Bug] 聚合模型BITMAP类型聚合结果不正确
URL: https://github.com/apache/doris/issues/28118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 聚合模型BITMAP类型聚合结果不正确 [doris]
Posted by "tangbinyeer (via GitHub)" <gi...@apache.org>.
tangbinyeer commented on issue #28118:
URL: https://github.com/apache/doris/issues/28118#issuecomment-1855555281
测试ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 聚合模型BITMAP类型聚合结果不正确 [doris]
Posted by "tangbinyeer (via GitHub)" <gi...@apache.org>.
tangbinyeer commented on issue #28118:
URL: https://github.com/apache/doris/issues/28118#issuecomment-1846452749
进一步测试,当写入的数据使用to_bitmap(int)时,结果仍然是错误的,且和上面的错误数据一致,hash记数仍然等于11,不像是随机的。

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 聚合模型BITMAP类型聚合结果不正确 [doris]
Posted by "LemonLiTree (via GitHub)" <gi...@apache.org>.
LemonLiTree commented on issue #28118:
URL: https://github.com/apache/doris/issues/28118#issuecomment-1849618874
2.0.3 没复现出来, 分批多次插入数据 是怎么做的?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 聚合模型BITMAP类型聚合结果不正确 [doris]
Posted by "tangbinyeer (via GitHub)" <gi...@apache.org>.
tangbinyeer commented on issue #28118:
URL: https://github.com/apache/doris/issues/28118#issuecomment-1849712591
错误版本是2.0.2,另外我在2.0.0版本做了测试,没有问题。
分批插入就是将数据集分成多个insert into而已。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 聚合模型BITMAP类型聚合结果不正确 [doris]
Posted by "LemonLiTree (via GitHub)" <gi...@apache.org>.
LemonLiTree commented on issue #28118:
URL: https://github.com/apache/doris/issues/28118#issuecomment-1851585248
等在2.0.3 发布后 你升级看看吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 聚合模型BITMAP类型聚合结果不正确 [doris]
Posted by "tangbinyeer (via GitHub)" <gi...@apache.org>.
tangbinyeer commented on issue #28118:
URL: https://github.com/apache/doris/issues/28118#issuecomment-1846472121
[doris分批插入bitmap聚合结果不正确.zip](https://github.com/apache/doris/files/13608406/doris.bitmap.zip)
这是完整的建表和插入语句,请帮忙分析下,谢谢!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org