You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/09 17:53:05 UTC
[GitHub] [arrow] alamb opened a new pull request #9968: ARROW-12267: [Rust] JSON writer does not support timestamp types
alamb opened a new pull request #9968:
URL: https://github.com/apache/arrow/pull/9968
# Rationale
I want to write timestamps to JSON and when I did so I got a panic :)
When I tried to write a`TimestampNanosecondArray` I got the following error:
```
thread 'influxdb_ioxd::http::tests::test_query_json' panicked at 'Unsupported datatype: Timestamp(
Nanosecond,
None,
)', /Users/alamb/.cargo/git/checkouts/arrow-3a9cfebb6b7b2bdc/3e825a7/rust/arrow/src/json/writer.rs:326:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
# Changes
Add support for DataType::Timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] houqp commented on pull request #9968: ARROW-12267: [Rust] Implement support for timestamps in JSON writer
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #9968:
URL: https://github.com/apache/arrow/pull/9968#issuecomment-817347780
The caller would need to obtain the response schema first, in most cases, the schema should be static. In case of API responses, it would be part of the API spec.
I would expect most of the users to use the string encoded format and only resort to numbers for performance critical use-cases. For example, in Airflow, by changing the column type from string encoded date time to timestamp numbers, we reduced the frontend load time by almost 90% :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9968: ARROW-12267: [Rust] Implement support for timestamps in JSON writer
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9968:
URL: https://github.com/apache/arrow/pull/9968#issuecomment-816870055
https://issues.apache.org/jira/browse/ARROW-12267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9968: ARROW-12267: [Rust] Implement support for timestamps in JSON writer
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9968:
URL: https://github.com/apache/arrow/pull/9968#issuecomment-817277440
> I am thinking of adding an option to return these type of values as numbers instead of human readable strings for better ser/de performance, what do you think about that?
@houqp I think an option for such behavior is a reasonable idea.
One question I have is if dates / timestamps / etc are produced as numbers, how would the other side know what type to use?
Like if we made JSON like this:
```
{
"name": "foo",
"time": 1234456788
}
```
Is `time` a `Timestamp`? A `Time`? a `u64`? something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9968: ARROW-12267: [Rust] Implement support for timestamps in JSON writer
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9968:
URL: https://github.com/apache/arrow/pull/9968#issuecomment-817813035
@houqp thanks -- that makes sense. Sending timestamps encoded as strings is definitely not a good idea for performance :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9968: ARROW-12267: [Rust] Implement support for timestamps in JSON writer
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9968:
URL: https://github.com/apache/arrow/pull/9968#issuecomment-816975255
FYI @houqp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb closed pull request #9968: ARROW-12267: [Rust] Implement support for timestamps in JSON writer
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9968:
URL: https://github.com/apache/arrow/pull/9968
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org