You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mightyshazam (via GitHub)" <gi...@apache.org> on 2023/10/20 18:04:00 UTC
[PR] Add optional handling for binary arrays in json [arrow-rs]
mightyshazam opened a new pull request, #4967:
URL: https://github.com/apache/arrow-rs/pull/4967
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #4945 .
# Rationale for this change
There is no native binary representation for json, so `arrow-json` simply returns an error when creating a decoder. The suggestion in the issue was to handle it manually, but there was no way to opt in to this behavior in the library. This allows a consumer to opt in to binary serialization, using their desired conversion method, or keep the default behavior.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
A user can implement the `BinaryDecoder` trait to
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
The addition of `pub fn with_binary_decoding(self, binary_decoder: BinaryDecoderProvider)` to the `ReaderBuilder` implementation in `arrow-json`.
New `BinaryArrayDecoder` trait in `arrow-json`.
<!---
If there are any breaking changes to public APIs, please add the `breaking change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add optional handling for binary arrays in json [arrow-rs]
Posted by "mightyshazam (via GitHub)" <gi...@apache.org>.
mightyshazam commented on PR #4967:
URL: https://github.com/apache/arrow-rs/pull/4967#issuecomment-1773463012
The example is a log file for a delta table. The log entries are in json and they contain a stats field that lists the min/max for each column in the schema. delta-rs writes the contents of binary columns as a serialized string using the below method
```
let escaped_bytes = v
.into_iter()
.flat_map(std::ascii::escape_default)
.collect::<Vec<u8>>();
let escaped_string = String::from_utf8(escaped_bytes).unwrap();
```
When creating a checkpoint, it takes that accumulated json from the various log files that looks something like this `{"add":{"path":"part-00001-8a7f1b6b-5869-4c37-8760-462ee9c97d49-c000.parquet","size":1522,"partitionValues":{},"modificationTime":1697658217285,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"tenant_model_id\":\"00000004\",\"content\":\"00000004\",\"job_id\":\"00000004\",\"platform_model_id\":\"00000004\"},\"maxValues\":{\"tenant_model_id\":\"00000004\",\"content\":\"00000004\",\"job_id\":\"00000004\",\"platform_model_id\":\"00000004\"},\"nullCount\":{\"job_id\":0,\"content\":0,\"tenant_model_id\":0,\"platform_model_id\":0}}"}}` where the `content` field is the binary field in this case.
The checkpoint code takes a series of json entries like the one above and uses the ReaderBuilder's `serialize` method to turn that into an RecordBatch since the checkpoints are in parquet format.
The issue isn't that the schema is in correct, but the usage may be. It might make sense to convert the delta schema to an arrow schema with utf-8 columns because that is what is ultimately written. However, the code as is just converts directly from a delta schema to an arrow schema and builds the decoder. Due to the lack of binary column support, the method returns an error instead of a decoder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add optional handling for binary arrays in json [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed pull request #4967: Add optional handling for binary arrays in json
URL: https://github.com/apache/arrow-rs/pull/4967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add optional handling for binary arrays in json [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #4967:
URL: https://github.com/apache/arrow-rs/pull/4967#issuecomment-1773417582
This might be a stupid question, but why does this need to be implemented in the JSON reader, as opposed to processing the StringArray after the fact?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add optional handling for binary arrays in json [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #4967:
URL: https://github.com/apache/arrow-rs/pull/4967#issuecomment-2657453316
IMO that older PR does _NOT_ solve the problem mentioned here. That PR gives a way to cast from binary to string (that happens to contain base64-encded binary data). But arrow JSON still doesn't have any actual support for encoding and decoding the arrow binary type.
That said, there's definitely an argument to be made that round trip encoding needs domain knowledge (ie does the sender and/or receiver expect hex? base64? ascii85? which variant? etc. So maybe manual casts are the best arrow itself can expose, to avoid getting sucked into that rabbit hole?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add optional handling for binary arrays in json [arrow-rs]
Posted by "mightyshazam (via GitHub)" <gi...@apache.org>.
mightyshazam commented on PR #4967:
URL: https://github.com/apache/arrow-rs/pull/4967#issuecomment-1773446489
> This might be a stupid question, but why does this need to be implemented in the JSON reader, as opposed to processing the StringArray after the fact?
This could all be the product of me not understanding my options. I started looking into this as part of work on the [delta-rs](https://github.com/delta-io/delta-rs) project. I was working on a kubernetes operator to do some maintenance when I encountered the ArrowJson error related to binary columns. It turns out we can't run checkpoints on tables with binary columns.
This was due to the call to [build_decoder](https://github.com/delta-io/delta-rs/blob/a9cdd605081a6e5ea5da7edff01f8d4bd3bfce77/rust/src/protocol/checkpoints.rs#L371C1-L373C27) in the `ReaderBuilder`. After a little research, I encountered the issue and your comments about handling it manually. Instead of creating a new reader for delta-rs, I thought it might be more reasonable to implement this as optional functionality in the arrow library.
I know it is probably a pretty niche case, but it was also a relatively simple way forward. If you can think of a better alternative, I am definitely open to that. This just turned out to be a pretty easy way forward.
Hopefully, that context makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add optional handling for binary arrays in json [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #4967:
URL: https://github.com/apache/arrow-rs/pull/4967#issuecomment-1773451728
Could you provide an example of what the contents of this binary file are encoded as? Assuming the data is valid JSON, they must be valid UTF-8, and can therefore be read as a regular StringArray, i.e. DataType::Utf8.
Now the question is what does the downstream expect, e.g. if the data in the JSON file is actually base64 encoded, do they expect a BinaryArray with this data decoded? Or is the fact the data is labelled binary just a misnomer and the data is actually UTF-8 but the charset has just been lost somewhere down the line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org