You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "KevinyhZou (via GitHub)" <gi...@apache.org> on 2025/01/17 08:58:28 UTC
[I] [CH]Optimize nested function calls [incubator-gluten]
KevinyhZou opened a new issue, #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557
### Description
In our online sqls, there are many nested functions calls, such as
- `select a.b.c from test_tbl` => `select get_struct_field((get_struct_field(a, b),c) from test_tbl`
- `select count(1) from test_tbl where a = 1 and b = 2 and c = 3` => `select count(1) from test_tbl where and(and(a = 1, b = 2), c = 3 )`
- `select count(1) from test_tbl where a = 1 or b = 2 or c = 3` => `select count(1) from test_tbl where or(or(a = 1, b = 2), c = 3 )`
.....
and these functions can be optimized by make the nested arguments collapsed , which means
`get_struct_field((get_struct_field(a, b),c)` => get_struct_field(a, b, c)
`and(and(a = 1, b = 2), c = 3 )` => and(a =1, b=2, c=3)
`or(or(a=1, b= 2), c=3)` => or(a=1, b=2, c=3)
which can reduce the function calls and improve performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2655743130
Is the rule in https://github.com/apache/incubator-gluten/pull/8305 disable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2655376434
1. get_json_object(get_json_object({"a":"{\\\"x\\\":5}"}', '$.a'), '$.x'), the json string has backslashes to escape quotes ;
2. get_json_object(get_json_object('{"a.b": 0}', '$.a), '$.b'), the json key contains dot character(.) and it's same as the collapsed json path;
you can also look at the comments at `CollapseGetJsonObjectExpressionRule`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2655374146
什么case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2655351139
改写成get_json_object(a, '$.b.c') , 有些case 兼容不了。 get_json_object(a, '$.b', '$.c') 不是spark `get_json_object` 兼容的形式，这里是通过ScalaUDF 的方式将get_json_object 转为 ScalaUDF，来兼容get_json_object(a, '$.b', '$.c') 这种形式。 @lgbo-ustc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2655745592
rule in #8305 is disable by default as the not passed case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2662858790
Hi @KevinyhZou, for `get_json_object(get_json_object('{"a.b": 0}', '$.a'), '$.b')`, we expect the result is still null after collapsed as `get_json_object('{"a.b": 0}', '$.a.b')`, right? I did a test in velox, it is null as expected (simdjson-based implementation). If CH got wrong result, I am wondering whether it is a bug in its native implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2655730634
The optimization for nested `get_json_object` should be put into a seperate issue, I think it will be a large change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li closed issue #8557: [CH]Optimize nested function calls
URL: https://github.com/apache/incubator-gluten/issues/8557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2662911919
Yes, it should be the problem of ch backend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2655674879
Let's make things clear.
We cannot know the type of one field of a json. It could be a object or string. When the field's type is string, the corrent results for following cases are
| query | result |
|:------|:--------:|
| get_json_object(get_json_object(s, '$.a'), '$.b') | something |
| get_json_object(s, '$.a.b') | null |
So we cannot use `get_json_object(s, '$.a.b')` to replace `get_json_object(get_json_object(s, '$.a'), '$.b')`. It will cause difference with vinilla for both following choices
- keep the behavior of `get_json_object(s, '$.a.b')` as before, but the result is different from `get_json_object(get_json_object(s, '$.a'), '$.b')`
- somehow let `get_json_object(s, '$.a.b')` have same result as `get_json_object(get_json_object(s, '$.a'), '$.b')`, but it will make the result of normal call `get_json_object(s, '$.a.b')` different from vanilla
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "KevinyhZou (via GitHub)" <gi...@apache.org>.
KevinyhZou commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2659024535
I have open a seperate PR for `get_json_object`: #8733 @lgbo-ustc , you can have a review at this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH]Optimize nested function calls [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on issue #8557:
URL: https://github.com/apache/incubator-gluten/issues/8557#issuecomment-2652752577
```
get_json_object(get_json_object(a, '$.b'), '$.c') => get_json_object(a, '$.b', '$.c')
```
印象之前是用spark rule改写成`get_json_object(a, '$.b.c')`, 这里为何改写成这样，这是spark function支持的形式？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org