You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/12/23 16:49:15 UTC
[I] Improve performace of `translate` expression [datafusion-comet]
andygrove opened a new issue, #2976:
URL: https://github.com/apache/datafusion-comet/issues/2976
### What is the problem the feature request solves?
`CometStringExpressionBenchmark` shows poor performance:
```
OpenJDK 64-Bit Server VM 11.0.22+7-LTS on Mac OS X 14.6.1
Apple M3 Max
translate: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 12 45 40 0.1 11898.4 1.0X
Comet (Scan) 14 82 39 0.1 13301.5 0.9X
Comet (Scan + Exec) 96 159 38 0.0 93534.9 0.1X
```
### Describe the potential solution
Review benchmark to make sure it is valid, then try and optimize.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve performance of `translate` expression [datafusion-comet]
Posted by "shuch3ng (via GitHub)" <gi...@apache.org>.
shuch3ng commented on issue #2976:
URL: https://github.com/apache/datafusion-comet/issues/2976#issuecomment-3693830780
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve performance of `translate` expression [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2976:
URL: https://github.com/apache/datafusion-comet/issues/2976#issuecomment-4397982500
```
Running benchmark: translate
Running case: Spark
Stopped after 146 iterations, 2006 ms
Running case: Comet (Scan)
Stopped after 145 iterations, 2001 ms
Running case: Comet (Scan + Exec)
Stopped after 655 iterations, 2003 ms
OpenJDK 64-Bit Server VM 17.0.17+10 on Mac OS X 26.3.1
Apple M3 Ultra
translate: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 13 14 1 0.1 12590.9 1.0X
Comet (Scan) 13 14 1 0.1 12674.8 1.0X
Comet (Scan + Exec) 2 3 1 0.4 2413.9 5.2X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve performance of `translate` expression [datafusion-comet]
Posted by "shuch3ng (via GitHub)" <gi...@apache.org>.
shuch3ng commented on issue #2976:
URL: https://github.com/apache/datafusion-comet/issues/2976#issuecomment-3693993973
Similarly to https://github.com/apache/datafusion-comet/issues/2975, I also couldn't reproduce the benchmark on the main branch
```
OpenJDK 64-Bit Server VM 17.0.17+10 on Mac OS X 26.2
Apple M2 Pro
translate: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 40808 41090 398 0.0 38917.7 1.0X
Comet (Scan) 41219 41246 39 0.0 39309.6 1.0X
Comet (Scan + Exec) 22084 22227 202 0.0 21060.8 1.8X
```
The physical plan is as below
```
== Physical Plan ==
*(1) CometColumnarToRow
+- CometProject [translate(col, abc, xyz)#3474], [translate(col#3473, abc, xyz) AS translate(col, abc, xyz)#3474]
+- CometScan [native_iceberg_compat] parquet spark_catalog.default.test[col#3473] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)
```
@coderfender could you please verify on your side? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve performance of `translate` expression [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2976: Improve performance of `translate` expression
URL: https://github.com/apache/datafusion-comet/issues/2976
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org