You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ahsanabbas123 (via GitHub)" <gi...@apache.org> on 2026/01/07 11:07:23 UTC

[I] [C++] Substrait serialised expression size increases exponentially due to extension URIs being repeated [arrow]

ahsanabbas123 opened a new issue, #48761:
URL: https://github.com/apache/arrow/issues/48761

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   When serialising pyarrow expressions using substrait, the size of the serialised buffer increases exponentially with the number of binary operation clauses in the expression due to extension URIs being repeated again for each nested expression.  
   
   Testing with pyarrow 21.0.0: 
   ```
   ================================================================================
   Testing with 1 OR condition(s)
   ================================================================================
   Expression:  (int_col == 1)
   Substrait size: 201 bytes
   Total extension URIs: 1
   
   ================================================================================
   Testing with 2 OR condition(s)
   ================================================================================
   Expression:  ((int_col == 1) or (int_col == 2))
   Substrait size: 634 bytes
   Total extension URIs: 5
   
   ================================================================================
   Testing with 4 OR condition(s)
   ================================================================================
   Expression:  ((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4))
   Substrait size: 2967 bytes
   Total extension URIs: 29
   
   ================================================================================
   Testing with 6 OR condition(s)
   ================================================================================
   Expression:  ((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6))
   Substrait size: 12017 bytes
   Total extension URIs: 125
   
   ================================================================================
   Testing with 8 OR condition(s)
   ================================================================================
   Expression:  ((((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6)) or (int_col == 7)) or (int_col == 8))
   Substrait size: 48306 bytes
   Total extension URIs: 509
   
   ================================================================================
   Testing with 10 OR condition(s)
   ================================================================================
   Expression:  ((((((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6)) or (int_col == 7)) or (int_col == 8)) or (int_col == 9)) or (int_col == 10))
   Substrait size: 193172 bytes
   Total extension URIs: 2045
   
   ================================================================================
   Testing with 12 OR condition(s)
   ================================================================================
   Expression:  ((((((((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6)) or (int_col == 7)) or (int_col == 8)) or (int_col == 9)) or (int_col == 10)) or (int_col == 11)) or (int_col == 12))
   Substrait size: 772342 bytes
   Total extension URIs: 8189
   
   ================================================================================
   Testing with 14 OR condition(s)
   ================================================================================
   Expression:  ((((((((((((((int_col == 1) or (int_col == 2)) or (int_col == 3)) or (int_col == 4)) or (int_col == 5)) or (int_col == 6)) or (int_col == 7)) or (int_col == 8)) or (int_col == 9)) or (int_col == 10)) or (int_col == 11)) or (int_col == 12)) or (int_col == 13)) or (int_col == 14))
   Substrait size: 3105111 bytes
   Total extension URIs: 32765
   
   ================================================================================
   Summary:
   ================================================================================
    OR Conditions  |  Size (bytes)   |    Size (KB)    | Extension URIs 
   ----------------------------------------------------------------------
          1        |       201       |      0.20       |        1       
          2        |       634       |      0.62       |        5       
          4        |      2,967      |      2.90       |       29       
          6        |     12,017      |      11.74      |       125      
          8        |     48,306      |      47.17      |       509      
         10        |     193,172     |     188.64      |      2045      
         12        |     772,342     |     754.24      |      8189      
         14        |    3,105,111    |     3032.33     |      32765     
   ```
    
   Serialization was done by simply: `buf = pas.serialize_expressions([expr], ["result"], schema)`
   
   Unfortunately, it might not be possible to just use `is_in` in the expression so that might not be a possible workaround. 
   
   Would be nice to know if I am somehow misusing the API which is causing this. Thanks! 
   
   cc: @westonpace 
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Substrait serialised expression size increases exponentially due to extension URIs being repeated [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #48761:
URL: https://github.com/apache/arrow/issues/48761#issuecomment-3721407336

   I've assigned you to this issue!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Substrait serialised expression size increases exponentially due to extension URIs being repeated [arrow]

Posted by "ahsanabbas123 (via GitHub)" <gi...@apache.org>.
ahsanabbas123 commented on issue #48761:
URL: https://github.com/apache/arrow/issues/48761#issuecomment-3719095757

   I suspect this could be due to serializing args here (which adds the URIs once): https://github.com/apache/arrow/blob/28855e0be2a5c43f4524e1130549c577f40da6da/cpp/src/arrow/engine/substrait/expression_internal.cc#L1401 
   
   and then again in `EncodeSubstraitCall` (adds URIs again): https://github.com/apache/arrow/blob/28855e0be2a5c43f4524e1130549c577f40da6da/cpp/src/arrow/engine/substrait/expression_internal.cc#L1267
   
   Would it make sense to do the remove the former and shift it inside each case `struct_field` / `list_element` / `if_else` / `is_in`  as a fix for this?   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++] Substrait serialised expression size increases exponentially due to extension URIs being repeated [arrow]

Posted by "mohit7705 (via GitHub)" <gi...@apache.org>.
mohit7705 commented on issue #48761:
URL: https://github.com/apache/arrow/issues/48761#issuecomment-3720711577

   Hi @ahsanabbas123  👋  
   
   I’d like to work on this issue if it’s available.  
   I have experience with C++ and debugging serialization / performance-related bugs, and I’m interested in investigating the exponential growth caused by repeated extension URIs.
   
   Please let me know if I can be assigned or if there are any specific guidelines I should follow before starting.
   
   Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org