You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zml1206 (via GitHub)" <gi...@apache.org> on 2024/09/10 05:57:46 UTC
[I] Offload complex type for parquet cause VeloxUserError [incubator-gluten]
zml1206 opened a new issue, #7178:
URL: https://github.com/apache/incubator-gluten/issues/7178
### Backend
VL (Velox)
### Bug description
Found in read delta lake with checkpoint in version 3.0.0rc1.
Simple to reproduce
```
val df = spark.range(10).toDF("aA")
.withColumn("a", struct("aA"))
df.write.mode("overwrite").parquet("tmp/t1")
spark.read.parquet("tmp/t1")
.union(spark.read.parquet("tmp/t1"))
.filter("a.aA > 3")
.collect()
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
org.apache.gluten.exception.GlutenException: Exception: VeloxUserError
Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: Field not found: aA. Available fields are: aa.
Retriable: False
Context: (n0_1).aA
Additional Context: Top-level Expression: and(isnotnull((n0_1).aA), greaterthan((n0_1).aA, 3:BIGINT))
Function: getChildIdx
File: /Users/zml/Desktop/git_hub/incubator-gluten/ep/build-velox/build/velox_ep/velox/type/Type.cpp
Line: 424
Stack trace:
# 0
# 1
# 2
# 3
# 4
# 5
# 6
# 7
# 8
# 9
# 10
# 11
# 12
# 13
# 14
# 15
# 16
# 17
# 18
# 19
# 20
# 21
# 22
# 23
# 24
# 25
# 26
# 27
# 28
# 29
# 30
# 31
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Offload struct type for parquet cause VeloxUserError [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on issue #7178:
URL: https://github.com/apache/incubator-gluten/issues/7178#issuecomment-2357312236
Then it should be problem of native parquet scan together with union, do you prefer to fix it by yourself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Offload struct type for parquet cause VeloxUserError [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE closed issue #7178: [VL] Offload struct type for parquet cause VeloxUserError
URL: https://github.com/apache/incubator-gluten/issues/7178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Offload struct type for parquet cause VeloxUserError [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #7178:
URL: https://github.com/apache/incubator-gluten/issues/7178#issuecomment-2354220947
@yma11 Any insights?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Offload complex type for parquet cause VeloxUserError [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on issue #7178:
URL: https://github.com/apache/incubator-gluten/issues/7178#issuecomment-2342560214
It cannot be reproduced without union or filter. I didn't find out what the problem is, can you help take a look @zhztheplayer @rui-mo thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Offload struct type for parquet cause VeloxUserError [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on issue #7178:
URL: https://github.com/apache/incubator-gluten/issues/7178#issuecomment-2357303473
Fallback scan or remove union, it also worked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Offload struct type for parquet cause VeloxUserError [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on issue #7178:
URL: https://github.com/apache/incubator-gluten/issues/7178#issuecomment-2357290601
> will it work if you use `a.aa`?
Yes, it worked with `a.aa`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Offload struct type for parquet cause VeloxUserError [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on issue #7178:
URL: https://github.com/apache/incubator-gluten/issues/7178#issuecomment-2357251754
> spark.read.parquet("tmp/t1")
> .union(spark.read.parquet("tmp/t1"))
> .filter("a.aA > 3")
> .collect()
will it work if you use `a.aa`?
```
spark.read.parquet("tmp/t1")
.union(spark.read.parquet("tmp/t1"))
.filter("a.aa > 3")
.collect()
```
I think the column name becomes lower case after some operation so it can't be matched any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org