You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "taepper (via GitHub)" <gi...@apache.org> on 2026/09/23 15:27:08 UTC
[PR] GH-51471: [C++] Disable simdjson threading in chunker [arrow]
taepper opened a new pull request, #51473:
URL: https://github.com/apache/arrow/pull/51473
### Rationale for this change
Resolves #51471:
When invoking `simdjson`'s `parse_many`, it uses threading to perform its stage 1 parsing on the next batch when while parsing the current batch:
https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md?plain=1#L96-L104
In most cases, we do not need to parse any batches after the first one to find delimiters (in fact, we currently would even error, when the first batch does not contain the whole first document). By disabling the parser's threading we can expect performance improvements
### What changes are included in this PR?
This disables threading in the parser by setting `parser_.threaded = false;`
### Are these changes tested?
Yes
before:
```
ChunkJSONPrettyPrintedMultipleBlocks 687263 ns 648184 ns 1099 block_size=27.344k
bytes_per_second=321.858Mi/s json_size=218.757k
```
after:
```
ChunkJSONPrettyPrintedMultipleBlocks 561073 ns 561058 ns 1309 block_size=27.344k
bytes_per_second=371.839Mi/s json_size=218.757k
```
only #51470:
```
ChunkJSONPrettyPrintedMultipleBlocks 197358 ns 167654 ns 3719 block_size=27.344k
bytes_per_second=1.2152Gi/s json_size=218.757k
```
together with #51470:
```
ChunkJSONPrettyPrintedMultipleBlocks 108151 ns 108148 ns 6463 block_size=27.344k
bytes_per_second=1.88383Gi/s json_size=218.757k
```
### Are there any user-facing changes?
No
### Was AI used for this PR?
**PR code and description written by:**
- [x] Human
- [ ] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51471: [C++] Disable simdjson threading in chunker [arrow]
Posted by "taepper (via GitHub)" <gi...@apache.org>.
taepper commented on PR #51473:
URL: https://github.com/apache/arrow/pull/51473#issuecomment-5869466122
> LGTM, can you rebase/merge so that we get updated CI?
Thank you for the review, and also #51472! I found these while trying to investigate possible improvements we can achieve with #51463. While studying the docs, I noticed that we definitely still lack some familiarity, and a few `simdjson` code sections that were recently introduced, use the library in suboptimal ways.
I merged main into the branch to retrigger CI with all latest changes (in particular including #51472)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51471: [C++] Disable simdjson threading in chunker [arrow]
Posted by "taepper (via GitHub)" <gi...@apache.org>.
taepper commented on PR #51473:
URL: https://github.com/apache/arrow/pull/51473#issuecomment-5876310674
> @taepper Can you perhaps update the PR description (especially benchmark numbers) now that https://github.com/apache/arrow/issues/51470 has been merged?
I updated the benchmarks, the PR description is still valid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51471: [C++] Disable simdjson threading in chunker [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #51473:
URL: https://github.com/apache/arrow/pull/51473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51471: [C++] Disable simdjson threading in chunker [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #51473:
URL: https://github.com/apache/arrow/pull/51473#issuecomment-5870335889
@taepper Can you perhaps update the PR description (especially benchmark numbers) now that https://github.com/apache/arrow/issues/51470 has been merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51471: [C++] Disable simdjson threading in chunker [arrow]
Posted by "taepper (via GitHub)" <gi...@apache.org>.
taepper commented on PR #51473:
URL: https://github.com/apache/arrow/pull/51473#issuecomment-5797692101
If you would have time for review, I think this is a good improvement to the chunker @pitrou (or @ others that might be a good fit to review this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51471: [C++] Disable simdjson threading in chunker [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51473:
URL: https://github.com/apache/arrow/pull/51473#issuecomment-5797657654
:warning: GitHub issue #51471 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org