You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/02 06:14:19 UTC
[GitHub] [arrow-rs] matthewmturner opened a new pull request #1383: Add write method to Json Writer
matthewmturner opened a new pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383
# Which issue does this PR close?
<!---
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #1340
# Rationale for this change
<!---
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
<!---
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are there any user-facing changes?
<!---
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] chmp commented on pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
chmp commented on pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383#issuecomment-1059750591
That sound's awesome could you point me to the impl for some inspiration? In case you're interested, I typed up an explanation for how `serde_arrow` works [here](https://github.com/chmp/serde_arrow/blob/main/serde_arrow/Implementation.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] chmp commented on pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
chmp commented on pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383#issuecomment-1059548659
Sure happy to sum up `serde_arrow`. First: at the moment it's only an experiement. I found it useful for some private data processing, and thought maybe it's also helpful to others. At the moment the status is usuable, but only those cases are handled, that I needed so far.
The basic idea is to allow record batches to be used as a "file format" for serde. A typical way of using serde is to take some Rust structs, to implement serialize, and then to generate JSON from these structs. `serde_arrow` allows you to use the same structs, but to generate record batches. So you don't have to use the `arrow` builder API, but can simply derive `serde::Serialize` and then call `serde_arrow::to_record_batch`. The reverse process, reading structs from a record batch is also supported. One complication is, that the data models of `serde` and `arrow` do not match. For that reason `serde_arrow` offers additional logic to let the user specify how to translate the `serde` data model into `arrow` one and back again.
For eaxmple:
```rust
#[derive(Serialize)]
struct Example {
a: f32,
b: i32,
}
let examples = vec![
Example { a: 1.0, b: 1 },
Example { a: 2.0, b: 2 },
];
let schema = serde_arrow::Schema::from_records(&examples)?;
let batch = serde_arrow::to_record_batch(&examples, schema)?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] v1gnesh commented on pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
v1gnesh commented on pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383#issuecomment-1058811849
Hello @alamb and @matthewmturner,
Any comments on how the work in these 2 projects overlaps with work on JSON/Arrow that happens here (in `arrow-rs`):
https://github.com/chmp/serde_arrow
https://github.com/simd-lite/simd-json-derive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] chmp commented on pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
chmp commented on pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383#issuecomment-1063210584
Thanks for the link. The code looks actually quite similar. Def. gonna borrow the idea to support iterables directly :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb merged pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
alamb merged pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383#issuecomment-1059597919
That is interesting -- @chmp -- I tried to do something similar with IOx (to create RecordBatches from vec's of objects we already had and then expose them as SQL queryable tables)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383#issuecomment-1063960836
Yeah, I would love a way to support automatically creating arrow arrays from something that implemented `serde` -- so many cool projects to do I just ran out of time ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383#issuecomment-1059753923
@chmp this is as far as I got: https://github.com/influxdata/influxdb_iox/issues/1013#issuecomment-806741682
We eventually went with manual conversion to Arrow :
https://github.com/influxdata/influxdb_iox/blob/main/db/src/system_tables.rs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] codecov-commenter commented on pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383#issuecomment-1056355667
# [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/1383?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1383](https://codecov.io/gh/apache/arrow-rs/pull/1383?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eb8bda4) into [master](https://codecov.io/gh/apache/arrow-rs/commit/483a5021bab6027821063c7dcaf5c0a0bd527d0e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (483a502) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head eb8bda4 differs from pull request most recent head bdfcc27. Consider uploading reports for the commit bdfcc27 to get more accurate results
[](https://codecov.io/gh/apache/arrow-rs/pull/1383?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #1383 +/- ##
==========================================
+ Coverage 83.00% 83.02% +0.01%
==========================================
Files 181 181
Lines 52994 53016 +22
==========================================
+ Hits 43990 44015 +25
+ Misses 9004 9001 -3
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/1383?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [arrow/src/json/writer.rs](https://codecov.io/gh/apache/arrow-rs/pull/1383/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2pzb24vd3JpdGVyLnJz) | `92.11% <100.00%> (+0.33%)` | :arrow_up: |
| [parquet\_derive/src/parquet\_field.rs](https://codecov.io/gh/apache/arrow-rs/pull/1383/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldF9kZXJpdmUvc3JjL3BhcnF1ZXRfZmllbGQucnM=) | `66.21% <0.00%> (+0.22%)` | :arrow_up: |
| [arrow/src/datatypes/field.rs](https://codecov.io/gh/apache/arrow-rs/pull/1383/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9maWVsZC5ycw==) | `54.10% <0.00%> (+0.30%)` | :arrow_up: |
| [arrow/src/datatypes/datatype.rs](https://codecov.io/gh/apache/arrow-rs/pull/1383/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9kYXRhdHlwZS5ycw==) | `66.80% <0.00%> (+0.39%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/1383?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/1383?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [483a502...bdfcc27](https://codecov.io/gh/apache/arrow-rs/pull/1383?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #1383: Add write method to Json Writer
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #1383:
URL: https://github.com/apache/arrow-rs/pull/1383#issuecomment-1059468873
Hi @v1gnesh I am not familiar with those crates. This is probably a question better directed at the authors of them (and if there is anything they wanted to contribute back to the arrow crate)
@chmp @Licenser
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org