You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/09 07:55:11 UTC
[GitHub] [incubator-doris] chaoyli opened a new pull request #3288: Fix bug
when use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
chaoyli opened a new pull request #3288: Fix bug when use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
URL: https://github.com/apache/incubator-doris/pull/3288
Now, column with REPLACE/REPLACE_IF_NOT_NULL can be filtered by ZoneMap/BloomFilter
when the rowset is base(version starts with zero). Always we think is an optimization.
But when some case, it will occurs bug.
```
create table test(
k1 int,
v1 int replace,
v2 int sum
);
```
If I have two records on different two versions
```
1 2 2 on version [0-10]
1 3 1 on version 11
```
If I perform a query select * from test where k1 = 1 and v1 = 3;
The result will be 1 3 1, this is not right because of the first record is filtered.
Remove this optimization is necessity to make the result is right.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #3288: Fix bug when use
ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
Posted by GitBox <gi...@apache.org>.
imay commented on issue #3288: Fix bug when use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
URL: https://github.com/apache/incubator-doris/pull/3288#issuecomment-611396794
If this optimization is removed, some query will suffer a performance degradation.
When the scanned value column type is all REPLACE, it seems that this optimization can be applied.
Can you leave this optimization for this case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #3288: Fix bug when
use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
Posted by GitBox <gi...@apache.org>.
chaoyli merged pull request #3288: Fix bug when use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
URL: https://github.com/apache/incubator-doris/pull/3288
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3288: Fix bug when
use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3288: Fix bug when use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
URL: https://github.com/apache/incubator-doris/pull/3288#issuecomment-611389566
#3289
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3288: Fix bug when
use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3288: Fix bug when use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
URL: https://github.com/apache/incubator-doris/pull/3288#issuecomment-611533083
> > > If this optimization is removed, some query will suffer a performance degradation.
> > > When the scanned value column type is all REPLACE, it seems that this optimization can be applied.
> > > Can you leave this optimization for this case?
> >
> >
> > I think it's a very corner case to select all columns on REPLACE and will waste much resource.
> >
> > 1. Only base rowset(version starts with zero) can use it, all other rowsets have to save ZoneMap/BloomFilter for REPLACE columns, which is non-sense.
> > 2. REPLACE is value column, the ZoneMap is not very useful because of sorted column is key column.
>
> Beside ZoneMap/BloomFilter, input filter will be applied before rowset streams are merged, this will cause this problem too. And in most cases, base rowset has the most rows, and filtering data before merging will reduce merge operation dramatically. It will result in a good performance.
> So I think ZoneMap and BloomFilter is not useful for REPLACE column, however filtering before merging worths it.
Year, filter not use it now, predicate only in Key column can be used to filtered. REPLACE column can be used ZoneMap and BloomFilter to filter base rowset.
For the above reasons, I think can remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #3288: Fix bug when use
ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
Posted by GitBox <gi...@apache.org>.
imay commented on issue #3288: Fix bug when use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
URL: https://github.com/apache/incubator-doris/pull/3288#issuecomment-611611537
> Year, filter not use it now, predicate only in Key column can be used to filtered. REPLACE column can be used ZoneMap and BloomFilter to filter base rowset.
> For the above reasons, I think can remove it.
Thanks for your reply, I see. After this CL, performance won't be influenced.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #3288: Fix bug when use
ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
Posted by GitBox <gi...@apache.org>.
imay commented on issue #3288: Fix bug when use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
URL: https://github.com/apache/incubator-doris/pull/3288#issuecomment-611468986
> > If this optimization is removed, some query will suffer a performance degradation.
> > When the scanned value column type is all REPLACE, it seems that this optimization can be applied.
> > Can you leave this optimization for this case?
>
> I think it's a very corner case to select all columns on REPLACE and will waste much resource.
>
> 1. Only base rowset(version starts with zero) can use it, all other rowsets have to save ZoneMap/BloomFilter for REPLACE columns, which is non-sense.
> 2. REPLACE is value column, the ZoneMap is not very useful because of sorted column is key column.
Beside ZoneMap/BloomFilter, input filter will be applied before rowset streams are merged, this will cause this problem too. And in most cases, base rowset has the most rows, and filtering data before merging will reduce merge operation dramatically. It will result in a good performance.
So I think ZoneMap and BloomFilter is not useful for REPLACE column, however filtering before merging worths it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3288: Fix bug when
use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3288: Fix bug when use ZoneMap/BloomFiter on column with REPLACE/REPLACE_IF_NOT_NULL
URL: https://github.com/apache/incubator-doris/pull/3288#issuecomment-611419945
> If this optimization is removed, some query will suffer a performance degradation.
> When the scanned value column type is all REPLACE, it seems that this optimization can be applied.
> Can you leave this optimization for this case?
I think it's a very corner case to select all columns on REPLACE and will waste much resource.
1. Only base rowset(version starts with zero) can use it, all other rowsets have to save ZoneMap/BloomFilter for REPLACE columns, which is non-sense.
2. REPLACE is value column, the ZoneMap is not very useful because of sorted column is key column.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org