You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Omega359 (via GitHub)" <gi...@apache.org> on 2025/01/20 16:31:45 UTC

[I] Potential performance improvement in contains and strpos functions [datafusion]

Omega359 opened a new issue, #14210:
URL: https://github.com/apache/datafusion/issues/14210

   ### Is your feature request related to a problem or challenge?
   
   I came across a [library](https://github.com/ashvardanian/StringZilla) that accelerates some string operations (primarily find for DataFusion's use case) using simd. It seems to be better optimized for larger strings than the memchr::memmem::find that is being used by arrow. For smaller strings on my machine the memchr crate seems to perform better but for longer strings stringzilla seems to outperform it quite significantly. The code changes required to support it are minimal though I would like to need performance comparisons on other architectures (I ran the following on my i9-13900h laptop)
   
   ```
   # cargo bench --bench contains --bench strpos -- --save-baseline main
   # <switch to stringzilla branch>
   # cargo bench --bench contains --bench strpos -- --save-baseline stringzilla
   # critcmp main stringzilla
   group                                          main                                    stringzilla
   -----                                          ----                                    -----------
   contains_StringArray_ascii_str_len_1024        32.47    16.1±0.95ms        ? ?/sec     1.00  496.2±114.11µs        ? ?/sec
   contains_StringArray_ascii_str_len_128         16.00     3.4±0.14ms        ? ?/sec     1.00   214.3±71.34µs        ? ?/sec
   contains_StringArray_ascii_str_len_32          1.00   185.0±24.42µs        ? ?/sec     1.01   187.2±20.89µs        ? ?/sec
   contains_StringArray_ascii_str_len_4096        15.92    50.1±5.63ms        ? ?/sec     1.00      3.1±0.26ms        ? ?/sec
   contains_StringArray_ascii_str_len_8           1.04     97.4±9.77µs        ? ?/sec     1.00     94.0±7.62µs        ? ?/sec
   contains_StringArray_utf8_str_len_1024         26.04    24.8±2.85ms        ? ?/sec     1.00  951.0±112.52µs        ? ?/sec
   contains_StringArray_utf8_str_len_128          14.07     5.1±0.44ms        ? ?/sec     1.00   362.2±85.63µs        ? ?/sec
   contains_StringArray_utf8_str_len_32           1.50   649.9±72.47µs        ? ?/sec     1.00   432.1±63.06µs        ? ?/sec
   contains_StringArray_utf8_str_len_4096         16.58    80.3±8.61ms        ? ?/sec     1.00      4.8±0.41ms        ? ?/sec
   contains_StringArray_utf8_str_len_8            1.00   236.2±29.07µs        ? ?/sec     1.08   254.4±16.59µs        ? ?/sec
   contains_StringViewArray_ascii_str_len_1024    31.25    16.1±0.85ms        ? ?/sec     1.00   515.8±99.46µs        ? ?/sec
   contains_StringViewArray_ascii_str_len_128     17.33     3.5±0.36ms        ? ?/sec     1.00   204.8±20.87µs        ? ?/sec
   contains_StringViewArray_ascii_str_len_32      1.06   192.6±27.58µs        ? ?/sec     1.00    182.2±9.15µs        ? ?/sec
   contains_StringViewArray_ascii_str_len_4096    14.90    49.0±4.56ms        ? ?/sec     1.00      3.3±0.42ms        ? ?/sec
   contains_StringViewArray_ascii_str_len_8       1.00     98.0±9.80µs        ? ?/sec     1.03    100.7±8.74µs        ? ?/sec
   contains_StringViewArray_utf8_str_len_1024     22.66    24.2±2.08ms        ? ?/sec     1.00  1068.3±369.70µs        ? ?/sec
   contains_StringViewArray_utf8_str_len_128      14.90     5.1±0.49ms        ? ?/sec     1.00   343.2±32.31µs        ? ?/sec
   contains_StringViewArray_utf8_str_len_32       1.49   623.7±43.61µs        ? ?/sec     1.00   417.7±31.81µs        ? ?/sec
   contains_StringViewArray_utf8_str_len_4096     15.93    79.7±8.05ms        ? ?/sec     1.00      5.0±0.63ms        ? ?/sec
   contains_StringViewArray_utf8_str_len_8        1.00   246.2±52.73µs        ? ?/sec     1.13   277.5±31.61µs        ? ?/sec
   strpos_StringArray_ascii_str_len_1024          9.96      9.8±1.15ms        ? ?/sec     1.00  986.1±116.33µs        ? ?/sec
   strpos_StringArray_ascii_str_len_128           3.39  1335.0±145.51µs        ? ?/sec    1.00   393.3±40.23µs        ? ?/sec
   strpos_StringArray_ascii_str_len_32            1.06   420.7±92.47µs        ? ?/sec     1.00   396.0±57.32µs        ? ?/sec
   strpos_StringArray_ascii_str_len_4096          5.85     40.6±3.68ms        ? ?/sec     1.00      6.9±0.84ms        ? ?/sec
   strpos_StringArray_ascii_str_len_8             1.07   150.4±12.31µs        ? ?/sec     1.00   141.0±10.02µs        ? ?/sec
   strpos_StringArray_utf8_str_len_1024           20.27    26.7±2.58ms        ? ?/sec     1.00  1316.5±127.39µs        ? ?/sec
   strpos_StringArray_utf8_str_len_128            6.71      3.9±0.27ms        ? ?/sec     1.00   586.3±88.39µs        ? ?/sec
   strpos_StringArray_utf8_str_len_32             2.45  1653.1±158.90µs        ? ?/sec    1.00   673.5±62.95µs        ? ?/sec
   strpos_StringArray_utf8_str_len_4096           14.90   101.7±4.95ms        ? ?/sec     1.00      6.8±0.79ms        ? ?/sec
   strpos_StringArray_utf8_str_len_8              1.82   713.9±45.84µs        ? ?/sec     1.00   392.3±44.63µs        ? ?/sec
   strpos_StringViewArray_ascii_str_len_1024      7.73      9.0±1.26ms        ? ?/sec     1.00  1167.2±114.81µs        ? ?/sec
   strpos_StringViewArray_ascii_str_len_128       2.59  1240.3±130.47µs        ? ?/sec    1.00   478.0±48.19µs        ? ?/sec
   strpos_StringViewArray_ascii_str_len_32        1.00   395.4±44.10µs        ? ?/sec     1.16   457.0±47.80µs        ? ?/sec
   strpos_StringViewArray_ascii_str_len_4096      4.42     36.2±2.29ms        ? ?/sec     1.00      8.2±1.16ms        ? ?/sec
   strpos_StringViewArray_ascii_str_len_8         1.00   192.9±14.14µs        ? ?/sec     1.06   204.2±18.46µs        ? ?/sec
   strpos_StringViewArray_utf8_str_len_1024       17.67    27.0±2.46ms        ? ?/sec     1.00  1526.3±205.96µs        ? ?/sec
   strpos_StringViewArray_utf8_str_len_128        6.85      4.1±0.40ms        ? ?/sec     1.00   593.2±51.54µs        ? ?/sec
   strpos_StringViewArray_utf8_str_len_32         2.49  1697.1±319.77µs        ? ?/sec    1.00   680.8±50.25µs        ? ?/sec
   strpos_StringViewArray_utf8_str_len_4096       13.95   103.3±7.00ms        ? ?/sec     1.00      7.4±0.91ms        ? ?/sec
   strpos_StringViewArray_utf8_str_len_8          1.93  776.5±127.84µs        ? ?/sec     1.00   401.6±78.46µs        ? ?/sec
   ```
   
   
   ### Describe the solution you'd like
   
   Incorporate the stringzilla into df functions where appropriate.
   
   ### Describe alternatives you've considered
   
   Leave the code as is.
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Potential performance improvement in contains and strpos functions [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 closed issue #14210: Potential performance improvement in contains and strpos functions
URL: https://github.com/apache/datafusion/issues/14210


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Potential performance improvement in contains and strpos functions [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #14210:
URL: https://github.com/apache/datafusion/issues/14210#issuecomment-2602849476

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org