You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "scovich (via GitHub)" <gi...@apache.org> on 2025/02/06 21:37:37 UTC

[PR] Expose record boundary information in JSON decoder [arrow-rs]

scovich opened a new pull request, #7092:
URL: https://github.com/apache/arrow-rs/pull/7092

   # Which issue does this PR close?
   
   Closes https://github.com/apache/arrow-rs/issues/6522
   
   # Rationale for this change
   
   The current JSON decoder has no way to distinguish record boundaries from buffer boundaries, which makes it very difficult to correctly and efficiently parse a series of unrelated JSON values (such as from a `StringArray` column). Bad examples include: blank strings (no rows produced), a single string containing multiple records (multiple rows produced), or multiple invalid strings whose concatenation looks like a single record (one row produced). 
   
   Such cases can be detected easily by checking the number of records parsed, and whether the last record was incomplete -- but that state is not publicly accessible (buried in the `TapeDecoder` struct).
   
   # What changes are included in this PR?
   
   Expose two new methods on the `TapeDecoder` struct, which support three new pub methods on `Decoder`, which exposes the number of records the decoder has buffered up so far, and whether the last record is partial (incomplete).
   
   Also update documentation and add unit tests.
   
   # Are there any user-facing changes?
   
   Two new public methods on `Decoder`: `has_partial_record`, `len`, and `is_empty`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Expose record boundary information in JSON decoder [arrow-rs]

Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #7092:
URL: https://github.com/apache/arrow-rs/pull/7092#issuecomment-2649013122

   > > Whilst this probably has some additional overheads, I'd be curious to see these quantified e.g. compared to the approach of not checking, I suspect these are low relative to the inherent costs of JSON decoding, and such an approach still benefits from the vectorised tape->array conversion.
   > 
   > In the common case where all strings contain correct JSON, the check should be branch-predicted away. It's ultimately just checking two variables that should already be hot in CPU cache, if not in registers, and both branches should be not-taken almost always.
   > 
   > In any case tho -- this enables the user of a `Decoder` to express correctness constraints they care about, and the small performance overhead would be totally acceptable. The change doesn't impact normal parsing at all.
   
   Actually... sending in a bunch of small (not I/O optimal) strings one at a time will probably be the biggest overhead. If we didn't need boundary validation, we could _probably_ just pass the entire underlying byte array from the `StringArray` in a single call. But that's unavoidable. Especially because I don't think the underlying byte array is required to be tightly packed (there could be regions of invalid bytes between strings).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Expose record boundary information in JSON decoder [arrow-rs]

Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #7092:
URL: https://github.com/apache/arrow-rs/pull/7092#issuecomment-2648951247

   > This makes sense to me, my understanding being this allows deserializing `StringArray` one value at a time, ensuring records are not split across value boundaries.
   
   That's a good description of what I hoped to achieve, yes.
   
   > Whilst this probably has some additional overheads, I'd be curious to see these quantified e.g. compared to the approach of not checking, I suspect these are low relative to the inherent costs of JSON decoding, and such an approach still benefits from the vectorised tape->array conversion.
   
   In the common case where all strings contain correct JSON, the check should be branch-predicted away. It's ultimately just checking two variables that should already be hot in CPU cache, if not in registers, and both branches should be not-taken almost always. 
   
   In any case tho -- this enables the user of a `Decoder` to express correctness constraints they care about, and the small performance overhead would be totally acceptable. The change doesn't impact normal parsing at all.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Expose record boundary information in JSON decoder [arrow-rs]

Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #7092:
URL: https://github.com/apache/arrow-rs/pull/7092#discussion_r1945502698


##########
arrow-json/src/reader/tape.rs:
##########
@@ -545,6 +545,16 @@ impl TapeDecoder {
         Ok(())
     }
 
+    /// The number of buffered rows, including the partially decoded row (if any).
+    pub fn num_buffered_rows(&self) -> usize {

Review Comment:
   For whatever reason, `TapeDecoder` seems to call them "rows" while `Decoder` calls them "records" so I named the new methods to match.



##########
arrow-json/src/reader/mod.rs:
##########
@@ -615,11 +615,27 @@ impl Decoder {
         self.tape_decoder.serialize(rows)
     }
 
+    /// True if the decoder is currently part way through decoding a record.
+    pub fn has_partial_record(&self) -> bool {
+        self.tape_decoder.has_partial_row()
+    }
+
+    /// The number of unflushed records, including the partially decoded record (if any).
+    pub fn len(&self) -> usize {
+        self.tape_decoder.num_buffered_rows()
+    }
+
+    /// True if there are no records to flush, i.e. [`len`] is zero.
+    pub fn is_empty(&self) -> bool {

Review Comment:
   Clippy prefers `is_empty()` over `len() == 0`, so I added both methods. I debated just exposing `num_buffered_records` instead, but len+is_empty seemed more rustic. 
   
   Happy to adjust the approach based on feedback.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Expose record boundary information in JSON decoder [arrow-rs]

Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #7092:
URL: https://github.com/apache/arrow-rs/pull/7092#discussion_r1945502698


##########
arrow-json/src/reader/tape.rs:
##########
@@ -545,6 +545,16 @@ impl TapeDecoder {
         Ok(())
     }
 
+    /// The number of buffered rows, including the partially decoded row (if any).
+    pub fn num_buffered_rows(&self) -> usize {

Review Comment:
   For whatever reason, `TapeDecoder` seems to call them "rows" while `Decoder` calls them "records" -- so I named the new methods to match.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Expose record boundary information in JSON decoder [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #7092:
URL: https://github.com/apache/arrow-rs/pull/7092


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Expose record boundary information in JSON decoder [arrow-rs]

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #7092:
URL: https://github.com/apache/arrow-rs/pull/7092#issuecomment-2649043911

   > If we didn't need boundary validation, we could probably just pass the entire underlying byte array
   
   Yes this was the approach I was comparing to, which of course would not work for strings with non-empty nulls, or malicious input, and so is probably not a good idea - I'm just curious 😅


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org