You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "taiyang-li (via GitHub)" <gi...@apache.org> on 2025/01/10 03:08:11 UTC
[I] [CH] sum with filter bad performance compared to vanilla spark [incubator-gluten]
taiyang-li opened a new issue, #8492:
URL: https://github.com/apache/incubator-gluten/issues/8492
### Backend
CH (ClickHouse)
### Bug description
gluten
```
0: jdbc:hive2://localhost:10000/> set spark.gluten.enabled = true;
+-----------------------+--------+
| key | value |
+-----------------------+--------+
| spark.gluten.enabled | true |
+-----------------------+--------+
1 row selected (0.034 seconds)
0: jdbc:hive2://localhost:10000/>
0: jdbc:hive2://localhost:10000/>
0: jdbc:hive2://localhost:10000/> select sum(if(id%3=0, id, 0)) from range(100000000);
+-----------------------------------+
| sum((IF(((id % 3) = 0), id, 0))) |
+-----------------------------------+
| 1666666683333333 |
+-----------------------------------+
1 row selected (64.729 seconds)
0: jdbc:hive2://localhost:10000/> select sum(if(id%3=0, id, 0)) from range(100000000);
+-----------------------------------+
| sum((IF(((id % 3) = 0), id, 0))) |
+-----------------------------------+
| 1666666683333333 |
+-----------------------------------+
1 row selected (64.811 seconds)
```
vanilla
```
0: jdbc:hive2://localhost:10000/> set spark.gluten.enabled = false;
+-----------------------+--------+
| key | value |
+-----------------------+--------+
| spark.gluten.enabled | false |
+-----------------------+--------+
1 row selected (0.09 seconds)
0: jdbc:hive2://localhost:10000/> select sum(id) filter(where id % 3 = 0) from range(100000000);
+----------------------------------------+
| sum(id) FILTER (WHERE ((id % 3) = 0)) |
+----------------------------------------+
| 1666666683333333 |
+----------------------------------------+
1 row selected (0.333 seconds)
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] sum with filter bad performance compared to vanilla spark [incubator-gluten]
Posted by "zhanglistar (via GitHub)" <gi...@apache.org>.
zhanglistar commented on issue #8492:
URL: https://github.com/apache/incubator-gluten/issues/8492#issuecomment-2655275193
> Update: after range operator is offloaded to CH. The performance of gluten is much faster, but still slower than vanilla spark (1.216s vs 0.333s).
>
> 0: jdbc:hive2://localhost:10000/> set spark.gluten.enabled = true;
> +-----------------------+--------+
> | key | value |
> +-----------------------+--------+
> | spark.gluten.enabled | true |
> +-----------------------+--------+
> 1 row selected (0.045 seconds)
> 0: jdbc:hive2://localhost:10000/> select sum(if(id%3=0, id, 0)) from range(100000000);
> +-----------------------------------+
> | sum((IF(((id % 3) = 0), id, 0))) |
> +-----------------------------------+
> | 1666666683333333 |
> +-----------------------------------+
> 1 row selected (1.216 seconds)
Second run, about 40% slower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] sum with filter bad performance compared to vanilla spark [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li closed issue #8492: [CH] sum with filter bad performance compared to vanilla spark
URL: https://github.com/apache/incubator-gluten/issues/8492
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [CH] sum with filter bad performance compared to vanilla spark [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on issue #8492:
URL: https://github.com/apache/incubator-gluten/issues/8492#issuecomment-2588875990
Update: after range operator is offloaded to CH. The performance of gluten is much faster, but still slower than vanilla spark.
``` sql
0: jdbc:hive2://localhost:10000/> set spark.gluten.enabled = true;
+-----------------------+--------+
| key | value |
+-----------------------+--------+
| spark.gluten.enabled | true |
+-----------------------+--------+
1 row selected (0.045 seconds)
0: jdbc:hive2://localhost:10000/> select sum(if(id%3=0, id, 0)) from range(100000000);
+-----------------------------------+
| sum((IF(((id % 3) = 0), id, 0))) |
+-----------------------------------+
| 1666666683333333 |
+-----------------------------------+
1 row selected (1.216 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org