You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by "emilk (via GitHub)" <gi...@apache.org> on 2025/09/15 15:04:03 UTC

[I] Improve `Display` for `DataType` [arrow-rs]

emilk opened a new issue, #8351:
URL: https://github.com/apache/arrow-rs/issues/8351

   The `Display` implementation for `DataType` is pretty bad for some things, including:
   
   * FixedSizeBinary
   * Timestamp
   * Struct (non-reversible / lossy)
   * Union
   * Dictionary
   * Decimal*
   * Map
   * RunEndEncoded
   
   We need to have a good overall design for these, and then implement it.
   
   ## Design considerations
   ### Readability
   The output should be short and readable for common cases, e.g. `List<nullable u8>`
   
   ### Reversibility
   We should be able to `parse` back the original `DataType`.
   
   Open question: Should that also include meta-data on any embedded `Field`:s?
   
   ### Safe
   Strings like field names need to be escaped to avoid string injection bugs (e.g. strings containing commas, quotes, newlines, …)
   
   ### Familiar
   We currently use parentheses for most datatypes, e.g. `List(u8)`. This is _fine_, but we should _consider_ using either `<>` like in C/Rust/C#/Java or even `[ ]` like in Python.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Improve `Display` for `DataType` [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.

Jefffrey commented on issue #8351:
URL: https://github.com/apache/arrow-rs/issues/8351#issuecomment-5405867402

   from a glance thats the only thing that stood out to me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Improve `Display` for `DataType` [arrow-rs]

Posted by "mhilton (via GitHub)" <gi...@apache.org>.

mhilton commented on issue #8351:
URL: https://github.com/apache/arrow-rs/issues/8351#issuecomment-3474175453

   I've just run into a problem which I think is close enough to this issue that it doesn't warrant opening a new one (If you disagree let me know and I'll create an issue). It turns out that the output of [`format_metadata`](https://github.com/apache/arrow-rs/blob/main/arrow-schema/src/datatype_display.rs#L24) is not deterministic. With the following reproducer
   
   ```rust
   use arrow::datatypes::{DataType, Field, Fields};
   use std::collections::HashMap;
   
   fn main() {
       println!(
           "{}",
           DataType::Struct(Fields::from(vec![
               Field::new("test1", DataType::Int32, true).with_metadata(HashMap::from([
                   ("a".to_string(), "A".to_string()),
                   ("b".to_string(), "B".to_string())
               ])),
               Field::new("test2", DataType::Utf8, false)
           ]))
       );
   }
   ```
   
   One either sees the output:
   
   ```
   Struct("test1": nullable Int32, metadata: {"a": "A", "b": "B"}, "test2": Utf8)
   ```
   
   or the output:
   
   ```
   Struct("test1": nullable Int32, metadata: {"b": "B", "a": "A"}, "test2": Utf8)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Improve `Display` for `DataType` [arrow-rs]

Posted by "emilk (via GitHub)" <gi...@apache.org>.

emilk commented on issue #8351:
URL: https://github.com/apache/arrow-rs/issues/8351#issuecomment-3414252017

   Looking over `datatype_display.rs`… not much.
   
   I think the formatting of `Decimal64` and friends is pretty bad though, e.g. `Decimal64(5, 6)` is not really understandable without memorizing the order of precision/scale. `Decimal64(precision: 5, scale: 6)` would be a lot better


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Improve `Display` for `DataType` [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.

etseidl commented on issue #8351:
URL: https://github.com/apache/arrow-rs/issues/8351#issuecomment-3559269434

   For any interested in this issue please check out #8890 (and #8883 which is the associated issue).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Improve `Display` for `DataType` [arrow-rs]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.

Jefffrey commented on issue #8351:
URL: https://github.com/apache/arrow-rs/issues/8351#issuecomment-5405735562

   i think there could still be some improvements here; for example, REE is still a bit ugly by default:
   
   ```
   RunEndEncoded("run_ends": non-null UInt32, "values": Int32)
   ```
   
   - consider dropping field names if its the default, similar to lists?
   - consider dropping the `non-null` as well since that will always be the case?
   
   though we can always file it as a separate issue and just close this epic


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org