You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "SteveLauC (via GitHub)" <gi...@apache.org> on 2023/04/26 02:59:58 UTC

[GitHub] [arrow-rs] SteveLauC opened a new issue, #4134: [arrow_json]`infer_json_schema()` infers `u64::MAX` as type `Int64`

SteveLauC opened a new issue, #4134:
URL: https://github.com/apache/arrow-rs/issues/4134

   **Describe the bug**
   [infer_json_schema()](https://docs.rs/arrow-json/latest/arrow_json/reader/fn.infer_json_schema.html) infers that `u64::MAX` is of type `Int64`, which is not correct as `i64` is not sufficient for `u64::MAX`.
   
   **To Reproduce**
   ```toml
   [package]
   name = "rust"
   version = "0.1.0"
   edition = "2021"
   
   [dependencies]
   arrow = "37.0.0"
   ```
   ```rust
   use arrow::json::reader::infer_json_schema;
   use std::io::BufReader;
   
   fn main() {
       let json = r#"{"uint64_max": 18446744073709551615 }"#;
       let mut buf_reader = BufReader::new(json.as_bytes());
       let schema = infer_json_schema(&mut buf_reader, Some(1)).unwrap();
   
       println!("{:?}", schema);
   }
   ```
   ```shell
   $ cargo r -q
   Schema { fields: [Field { name: "uint64_max", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {} }
   ```
   
   
   **Expected behavior**
   `u64::MAX` should be inferred as type `UInt64`
   
   **Additional context**
   Related code snippet: https://github.com/apache/arrow-rs/blob/b9819219c8fb6c7e5c7336e3c2fcd86cb5befd98/arrow-json/src/reader/schema.rs#L444-L451
   
   We can see that it defaults to `Int64` as long  as it is not of type `f64`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold commented on issue #4134: [arrow_json]`infer_json_schema()` infers `u64::MAX` as type `Int64`

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4134:
URL: https://github.com/apache/arrow-rs/issues/4134#issuecomment-1523202970

   I think the logic should probably be
   
   ```
    if n.is_i64() { 
        set_object_scalar_field_type(field_types, k, DataType::Int64)?; 
    } else { 
        set_object_scalar_field_type(field_types, k, DataType::Float64)?; 
    }
   ```
   
   Adding inference for unsigned integers would be a breaking change. I would be happy to review a PR that made this change


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold commented on issue #4134: [arrow_json]`infer_json_schema()` infers `u64::MAX` as type `Int64`

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4134:
URL: https://github.com/apache/arrow-rs/issues/4134#issuecomment-1523251890

   I had been hoping to unify the JSON and CSV schema inference as part of #4132, however, the CSV logic similarly does not currently perform any range checking #2580. I'll see what I can come up with


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] SteveLauC commented on issue #4134: [arrow_json]`infer_json_schema()` infers `u64::MAX` as type `Int64`

Posted by "SteveLauC (via GitHub)" <gi...@apache.org>.
SteveLauC commented on issue #4134:
URL: https://github.com/apache/arrow-rs/issues/4134#issuecomment-1523239786

   > Adding inference for unsigned integers would be a breaking change. 
   Agree. Can we add it in the next major release?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] SteveLauC commented on issue #4134: [arrow_json]`infer_json_schema()` infers `u64::MAX` as type `Int64`

Posted by "SteveLauC (via GitHub)" <gi...@apache.org>.
SteveLauC commented on issue #4134:
URL: https://github.com/apache/arrow-rs/issues/4134#issuecomment-1522702452

   Happy to submit a PR to get it fixed if you guys want to:)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org