You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2024/10/31 06:50:15 UTC
[I] [VL][1.2] Cast string to integral type does not ignore ISO control characters [incubator-gluten]
wForget opened a new issue, #7749:
URL: https://github.com/apache/incubator-gluten/issues/7749
### Backend
VL (Velox)
### Bug description
1. create test dataset
```
create table t1 as select url_decode('111111%00') as c1;
```
2. cast to integral type
check sqls:
```
select cast(c1 as int) from t1;
select cast(c1 as bigint) from t1;
```
gluten 1.2 + spark 3.5:
```
NULL
NULL
```
vanilla spark:
```
111111
111111
```
3. cast to fractional type
check sqls:
```
select cast(c1 as float) from t1;
select cast(c1 as double) from t1;
```
gluten 1.2 + spark 3.5:
```
111111.0
111111.0
```
vanilla spark:
```
111111.0
111111.0
```
Related PRs: https://github.com/apache/spark/pull/41535
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2][Result mismatch] Cast string to integral type does not ignore ISO control characters [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE closed issue #7749: [VL][1.2][Result mismatch] Cast string to integral type does not ignore ISO control characters
URL: https://github.com/apache/incubator-gluten/issues/7749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2][Result mismatch] Cast string to integral type does not ignore ISO control characters [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #7749:
URL: https://github.com/apache/incubator-gluten/issues/7749#issuecomment-2449175462
I will try to fix it, it seems to be related to this code:
https://github.com/facebookincubator/velox/blob/main/velox/functions/sparksql/specialforms/SparkCastHooks.cpp#L70-L75
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2][Result mismatch] Cast string to integral type does not ignore ISO control characters [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #7749:
URL: https://github.com/apache/incubator-gluten/issues/7749#issuecomment-2454697462
> String type conversions were also fixed in #1569 . They used trim to remove invalid characters, but that didn't handle invalid characters inside a String. It would be great to fix this in velox so we can remove #1569 in gluten. cc @PHILO-HE
Thank you for the information. Since the trim behavior for different types of cast in spark is different, it seems that extending #1569 is a simpler solution, can we fix it that way first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2][Result mismatch] Cast string to integral type does not ignore ISO control characters [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on issue #7749:
URL: https://github.com/apache/incubator-gluten/issues/7749#issuecomment-2449191386
String type conversions were also fixed in #1569 . They used trim to remove invalid characters, but that didn't handle invalid characters inside a String. It would be great to fix this in velox so we can remove #1569 in gluten.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org