You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/12/23 16:49:15 UTC

[I] Improve performace of `translate` expression [datafusion-comet]

andygrove opened a new issue, #2976:
URL: https://github.com/apache/datafusion-comet/issues/2976

   ### What is the problem the feature request solves?
   
   `CometStringExpressionBenchmark` shows poor performance:
   
   ```
   OpenJDK 64-Bit Server VM 11.0.22+7-LTS on Mac OS X 14.6.1
   Apple M3 Max
   translate:                                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   Spark                                                12             45          40          0.1       11898.4       1.0X
   Comet (Scan)                                         14             82          39          0.1       13301.5       0.9X
   Comet (Scan + Exec)                                  96            159          38          0.0       93534.9       0.1X
   ```
   
   ### Describe the potential solution
   
   Review benchmark to make sure it is valid, then try and optimize.
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Improve performance of `translate` expression [datafusion-comet]

Posted by "shuch3ng (via GitHub)" <gi...@apache.org>.
shuch3ng commented on issue #2976:
URL: https://github.com/apache/datafusion-comet/issues/2976#issuecomment-3693830780

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Improve performance of `translate` expression [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2976:
URL: https://github.com/apache/datafusion-comet/issues/2976#issuecomment-4397982500

   ```
   Running benchmark: translate
     Running case: Spark
     Stopped after 146 iterations, 2006 ms
     Running case: Comet (Scan)
     Stopped after 145 iterations, 2001 ms
     Running case: Comet (Scan + Exec)
     Stopped after 655 iterations, 2003 ms
   
   OpenJDK 64-Bit Server VM 17.0.17+10 on Mac OS X 26.3.1
   Apple M3 Ultra
   translate:                                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   Spark                                                13             14           1          0.1       12590.9       1.0X
   Comet (Scan)                                         13             14           1          0.1       12674.8       1.0X
   Comet (Scan + Exec)                                   2              3           1          0.4        2413.9       5.2X
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Improve performance of `translate` expression [datafusion-comet]

Posted by "shuch3ng (via GitHub)" <gi...@apache.org>.
shuch3ng commented on issue #2976:
URL: https://github.com/apache/datafusion-comet/issues/2976#issuecomment-3693993973

   Similarly to https://github.com/apache/datafusion-comet/issues/2975, I also couldn't reproduce the benchmark on the main branch
   ```
   OpenJDK 64-Bit Server VM 17.0.17+10 on Mac OS X 26.2
   Apple M2 Pro
   translate:                                Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   Spark                                             40808          41090         398          0.0       38917.7       1.0X
   Comet (Scan)                                      41219          41246          39          0.0       39309.6       1.0X
   Comet (Scan + Exec)                               22084          22227         202          0.0       21060.8       1.8X
   ```
   
   The physical plan is as below
   ```
   == Physical Plan ==
   *(1) CometColumnarToRow
   +- CometProject [translate(col, abc, xyz)#3474], [translate(col#3473, abc, xyz) AS translate(col, abc, xyz)#3474]
      +- CometScan [native_iceberg_compat] parquet spark_catalog.default.test[col#3473] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)
   ```
   
   @coderfender could you please verify on your side? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Improve performance of `translate` expression [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2976: Improve performance of `translate` expression
URL: https://github.com/apache/datafusion-comet/issues/2976


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org