You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "FelixYBW (via GitHub)" <gi...@apache.org> on 2024/07/26 01:53:32 UTC
[I] [VL] parquet write error [incubator-gluten]
FelixYBW opened a new issue, #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600
### Backend
VL (Velox)
### Bug description
```
24/07/26 00:58:09 ERROR [Driver] datasources.FileFormatWriter: Aborting job c89ecb16-71d7-49b6-8963-cd58dd9873e1.
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object
at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:160)
at org.apache.spark.sql.types.StructType$.$anonfun$fromAttributes$1(StructType.scala:548)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:238)
at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
```
the write columns are all string and bigint. except one date column which is used as partition column.
@JkSelf is the date columun the reason?
### Spark version
Spark-3.2.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2257575073
Can you assign this issue to me, I want to try to solve it, thank you. @FelixYBW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] parquet write error [incubator-gluten]
Posted by "LoseYSelf (via GitHub)" <gi...@apache.org>.
LoseYSelf commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2252045621
what is the sql and the table metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2283007546
> Do you have time to look at this issue? If not, I can follow up. Thanks.
I'll try to reproduce and solve it. @JkSelf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2257767183
I have disabled the date type in https://github.com/apache/incubator-gluten/pull/6637. @zml1206, if you have the time, you are welcome to continue supporting the date type in window range frame. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2287904108
fixed by #6803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf closed issue #6600: [VL] window function error
URL: https://github.com/apache/incubator-gluten/issues/6600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2258760408
@JkSelf Is there dependency of your current streaming window PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2282999717
@FelixYBW I tested the fix in my local environment and confirmed that the issue has been resolved. However, I encountered the following exception during execution `select sum(impression_count_last7days ) from ( SELECT sum(impression_count) OVER ( PARTITION BY user_id, surface ORDER BY date RANGE BETWEEN 6 preceding AND CURRENT ROW ) AS impression_count_last7days from d233 )"`
```
24/07/26 00:58:09 ERROR [Driver] datasources.FileFormatWriter: Aborting job c89ecb16-71d7-49b6-8963-cd58dd9873e1.
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object
at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:160)
at org.apache.spark.sql.types.StructType$.$anonfun$fromAttributes$1(StructType.scala:548)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:238)
at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
```
@zml1206 Do you have time to look at this issue? If not, I can follow up. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2278957365
@JkSelf Looks the issue isn't fixed by 6600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2253857074
The error is caused by window function:
```
sum(a) OVER (
PARTITION BY b,
c
ORDER BY date RANGE BETWEEN 6 preceding AND CURRENT ROW
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW closed issue #6600: [VL] window function error
URL: https://github.com/apache/incubator-gluten/issues/6600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2278962600
https://github.com/apache/incubator-gluten/blob/51d0a37faafc0457d594d180f7787e8c0da1f518/backends-velox/src/test/scala/org/apache/gluten/execution/TestOperator.scala#L377
The UT verification here has passed. Are there any special circumstances that have been missed? Can you provide the test sql? @FelixYBW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] window function error [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6600:
URL: https://github.com/apache/incubator-gluten/issues/6600#issuecomment-2279548682
@JkSelf Did you verify the SQL I shared? failed from my test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org