You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "davlee1972 (via GitHub)" <gi...@apache.org> on 2025/06/20 18:39:20 UTC
[I] Better Snowflake support for mapping snowflake Ints and Floats to Arrow/Pandas/Polars [arrow-adbc]
davlee1972 opened a new issue, #3008:
URL: https://github.com/apache/arrow-adbc/issues/3008
### What feature or improvement would you like to see?
https://docs.snowflake.com/en/sql-reference/data-types-numeric
According to their own documentation..
But it doesn't look like ADBC is using this info and sizing everything to the internal decimal(38,0) type Snowflake uses.
In reality Snowflake doesn't use a decimal(38,0) either. It finds the largest INT value in a micropartition and sizes that entire micropartiion Int column to 128, 64, 32, 16 or 8 bit integers..
Shouldn't the ADBC Driver look at the metadata and if a column is defined as a TinyInt it should allocate a PyArrow Int8 instead of a Int128.. This would save a lot of memory on the client side..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Better Snowflake support for mapping snowflake Ints and Floats to Arrow/Pandas/Polars [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #3008:
URL: https://github.com/apache/arrow-adbc/issues/3008#issuecomment-2992566003
The problem is that every microbatch can return a different width, and Snowflake doesn't track the maximum width used across the entire table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Better Snowflake support for mapping snowflake Ints and Floats to Arrow/Pandas/Polars [arrow-adbc]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on issue #3008:
URL: https://github.com/apache/arrow-adbc/issues/3008#issuecomment-2993805662
> Shouldn't the ADBC Driver look at the metadata and if a column is defined as a TinyInt
The problem is that Snowflake doesn't remember that you used the alias to create the type, so whether you created the column as TINYINT, INT, BIGINT or NUMBER(38, 0), it looks identical in the metadata. I can see what kind of advantages this might have, but from the perspective of generic client consumption it feels considerably less than ideal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org