You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/12/16 15:44:55 UTC

[I] Regression in struct casting in 57.2.0 (not yet released) [arrow-rs]

alamb opened a new issue, #9005:
URL: https://github.com/apache/arrow-rs/issues/9005

   **Describe the bug**
   I found this while testing arrow with DataFusion:
   - https://github.com/apache/datafusion/pull/19355
   
   - Related to https://github.com/apache/arrow-rs/issues/8465
   
   There are several queries like this in the DataFusion tests that work with 57.1.0 but do not
   work with main (what will be 57.2.0):
   
   ```sql
   CREATE TABLE struct_values (
       s1 struct<INT>,
       s2 struct<a INT,b VARCHAR>
   ) AS VALUES
     (struct(1), struct(1, 'string1')),
     (struct(2), struct(2, 'string2')),
     (struct(3), struct(3, 'string3'))
   ;
   ```
   
   They fail with this error
   
   >  DataFusion error: Execution error: type mismatch and can't cast to got Struct("c0": Int64, "c1": Utf8) and Struct("a": Int32, "b": Utf8View)
   
   
   **Expected behavior**
   The tests should pass as they did previously
   
   
   **Additional context**
   
   The change came in via  7e637a7559837b5a0171b23469e9652f2f83364b / https://github.com/apache/arrow-rs/pull/8871 from @brancz
   
   The tests pass when I revert this commit:
   
   ```shell
   git revert 7e637a7559837b5a0171b23469e9652f2f83364b
   ```
   
   I believe what is going on is that the `struct`  function creates types with placeholder field names (c0, c1, etc). You can see it like this
   
   ```sql
   > select arrow_typeof(struct(1));
   +--------------------------------+
   | arrow_typeof(struct(Int64(1))) |
   +--------------------------------+
   | Struct("c0": Int64)            |
   +--------------------------------+
   1 row(s) fetched.
   Elapsed 0.029 seconds.
   ```
   
   ```sql
   > select arrow_typeof(struct(1, 'string1'));
   +------------------------------------------------+
   | arrow_typeof(struct(Int64(1),Utf8("string1"))) |
   +------------------------------------------------+
   | Struct("c0": Int64, "c1": Utf8)                |
   +------------------------------------------------+
   ```
   
   In prior versions of arrow, the fields could be matched by position, but now
   they are matched by name and thus the above queries fail.
   
   
   
   A suggested solution would be to try and match by name first, and if that fails, try and match by position.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Regression in struct casting in 57.2.0 (not yet released) [arrow-rs]

Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #9005:
URL: https://github.com/apache/arrow-rs/issues/9005#issuecomment-3662090772

   Here it is: https://github.com/apache/arrow-rs/pull/9007
   
   After checking the patch again, I think I could be ok with this after all, since this only ever worked if and only if the number of fields in the source and destination struct are identical. I think using the struct building functions would be more explicit, but I don't feel that strongly after all about cast potentially being capable of this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Regression in struct casting in 57.2.0 (not yet released) [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9005:
URL: https://github.com/apache/arrow-rs/issues/9005#issuecomment-3662051263

   > Ack. Working on a fix.
   
   Thank you 🙏 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Regression in struct casting in 57.2.0 (not yet released) [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9005:
URL: https://github.com/apache/arrow-rs/issues/9005#issuecomment-3661759175

   > Aha, looking at the test, yes we could just use `named_struct` to build it. 
   
   
   > That said, if you feel strongly, I can prepare a patch that checks whether fields can successfully be casted in-order and perform that, which wouldn't be terribly difficult to do.
   
   I am not sure I feel "strongly" per se in general  -- keeping the existing behavior for the arrow patch release (so that queries that used to work keep working) I do think is pretty important.
   
   Perhaps we can consider a more nuanced fix for the next major release (e.g fix it downstream in DataFusion)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Regression in struct casting in 57.2.0 (not yet released) [arrow-rs]

Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #9005:
URL: https://github.com/apache/arrow-rs/issues/9005#issuecomment-3661725824

   If I'm understanding this right, the idea is that this is to rename a field? That seems like an odd use case for casting, wouldn't a better thing be to use the struct-building functionality, like [`named_struct`](https://datafusion.apache.org/user-guide/sql/scalar_functions.html#named-struct) and build it from the "previous" struct? If casting did this, I think it's far too easy to do the wrong thing accidentally.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Regression in struct casting in 57.2.0 (not yet released) [arrow-rs]

Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #9005:
URL: https://github.com/apache/arrow-rs/issues/9005#issuecomment-3662031256

   Ack. Working on a fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Regression in struct casting in 57.2.0 (not yet released) [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9005:
URL: https://github.com/apache/arrow-rs/issues/9005#issuecomment-3716629263

   `label_issue.py` automatically added labels {'arrow'} from #9007


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Regression in struct casting in 57.2.0 (not yet released) [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #9005: Regression in struct casting in 57.2.0 (not yet released)
URL: https://github.com/apache/arrow-rs/issues/9005


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Regression in struct casting in 57.2.0 (not yet released) [arrow-rs]

Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #9005:
URL: https://github.com/apache/arrow-rs/issues/9005#issuecomment-3661744641

   Aha, looking at the test, yes we could just use `named_struct` to build it. That said, if you feel strongly, I can prepare a patch that checks whether fields can successfully be casted in-order and perform that, which wouldn't be terribly difficult to do.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org