You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "lwz9103 (via GitHub)" <gi...@apache.org> on 2024/08/27 09:02:27 UTC
[I] [CH] Incorrect result when filter timestamp column with string values. [incubator-gluten]
lwz9103 opened a new issue, #7032:
URL: https://github.com/apache/incubator-gluten/issues/7032
### Backend
CH (ClickHouse)
### Bug description
```
create table filter_test (account_id int, record_time date, account_user_country string, account_date date, account_time timestamp);
insert into filter_test values
(20201001, 2020-10-01, shanghai, 2020-10-01, 2020-10-01 10:10:10),
(20201005, 2020-10-05, shanghai, 2020-10-05, 2020-10-05 10:10:10),
(20201008, 2020-10-08, beijing, 2020-10-08, 2020-10-08 10:10:10);
select * from filter_timestamp where account_time in ('2020-10-01 10:10:10', '2020-10-01 10:10:11')
```
gluten result is empty but spark result has 1 row.
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Incorrect result when filter timestamp column with string values. [incubator-gluten]
Posted by "lwz9103 (via GitHub)" <gi...@apache.org>.
lwz9103 commented on issue #7032:
URL: https://github.com/apache/incubator-gluten/issues/7032#issuecomment-2311979648
### Root Cause
If the data type of the filter column is not the same as the data type of the filter value, Spark will cast the left-side expression.
For example, where t3 in ('2020-10-01 10:10:10', '2020-10-01 10:10:11') will be converted to where cast(t3 as string) in ('2020-10-01 10:10:10', '2020-10-01 10:10:11').
However, in Gluten Clickhouse, Spark's timestamp type is mapped to Clickhouse's datetime64(6) type.
So when casting the timestamp column to a string, the value will become '2020-10-01 10:10:10.000000'.
This is why the filter condition doesn't take effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Incorrect result when in-filter timestamp column with string values. [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen closed issue #7032: [CH] Incorrect result when in-filter timestamp column with string values.
URL: https://github.com/apache/incubator-gluten/issues/7032
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] Incorrect result when in-filter timestamp column with string values. [incubator-gluten]
Posted by "lwz9103 (via GitHub)" <gi...@apache.org>.
lwz9103 commented on issue #7032:
URL: https://github.com/apache/incubator-gluten/issues/7032#issuecomment-2312039344
### Workaround
in-filter values use exact datatype
```
select * from filter_timestamp where account_time in (timestamp'2020-10-01 10:10:10', timestamp'2020-10-01 10:10:11')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org