You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/02/13 14:04:44 UTC
[I] Cast string to timestamp not compatible with Spark logic [arrow-datafusion-comet]
andygrove opened a new issue, #14:
URL: https://github.com/apache/arrow-datafusion-comet/issues/14
I was manually experimenting with some cast operations based on my experience of implementing them in Spark RAPIDS and found the following example of incorrect behavior. I would recommend implementing some fuzz tests to find these kind of issues.
## Test data
```
scala> robots.show
+------+
| name|
+------+
|WALL-E|
| R2D2|
| T2|
+------+
```
## Test with Comet
```
scala> import org.apache.spark.sql.types._
scala> val df = robots.withColumn("date", col("name").cast(DataTypes.TimestampType))
scala> df.show
+------+----+
| name|date|
+------+----+
|WALL-E|null|
| R2D2|null|
| T2|null|
+------+----+
```
## Test with Spark
```
scala> spark.conf.set("spark.comet.enabled", false)
scala> df.show
+------+-------------------+
| name| date|
+------+-------------------+
|WALL-E| null|
| R2D2| null|
| T2|2024-02-09 02:00:00|
+------+-------------------+
```
`T2` is a valid timestamp because `T` is the separator between the optional date and the time portion. `2` is a valid time because some time fields are optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cast string to timestamp not compatible with Spark logic [arrow-datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on issue #14:
URL: https://github.com/apache/arrow-datafusion-comet/issues/14#issuecomment-1942460969
cc @parthchandra this might be of interest to you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cast string to timestamp not compatible with Spark logic [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #14:
URL: https://github.com/apache/datafusion-comet/issues/14#issuecomment-2079763001
Closing this because there is a new more detailed issue for this:
https://github.com/apache/datafusion-comet/issues/328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Cast string to timestamp not compatible with Spark logic [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #14: Cast string to timestamp not compatible with Spark logic
URL: https://github.com/apache/datafusion-comet/issues/14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org