You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "wanweiqiangintel (via GitHub)" <gi...@apache.org> on 2023/05/06 07:28:33 UTC
[GitHub] [arrow] wanweiqiangintel opened a new issue, #35460: can we use simdjson to replace rapidjson
wanweiqiangintel opened a new issue, #35460:
URL: https://github.com/apache/arrow/issues/35460
### Describe the enhancement requested
As the performance result mentioned in simdjson community: the simdjson library uses three-quarters less instructions than state-of-the-art parser RapidJSON. And the throughput of simdjson is much higher than that of rapidjson:
So can we replace rapidjson with simdjson to implement json parser?
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-3114230912
I've started looking at this, but found that SIMDJSON has very little support for serialization (see the upstream [discussion](https://github.com/simdjson/simdjson/discussions/2086))
It looks like some initial work to implement serialization support was merged just [last week](https://github.com/simdjson/simdjson/pull/2282), but that requires C++26 and hasn't made its way into an official release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2797190801
For what it's worth, I think DuckDB uses yyjson, although I don't know if simdjson was considered. I seem to remember some past issues with homebrew packaging with respect to buildtime vs. runtime SIMD support (although I imagine simdjson has considered this as well!).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "lemire (via GitHub)" <gi...@apache.org>.
lemire commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-3262366875
We will be presenting our recent work at cppcon 2025: https://cppcon2025.sched.com/event/27bQx/reflection-based-json-in-c++-at-gigabytes-per-second
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2809849645
I agree we need to do something about our RapidJSON dependency. simdjson is a reasonable contender. We should just have to check it doesn't reduce the current platform compatibility (especially when SIMD isn't available/supported by simdjson).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #35460: can we use simdjson to replace rapidjson
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-1545741395
Seems that writer can still use original logic, but parser can make full use of simdjson?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-5411777827
> Is this something one of these issues will address or should we open a new one?
@Krishnanand-G is working on the issue, we can try the approach suggested by @pitrou https://github.com/apache/arrow/pull/50900#issuecomment-5411157545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #35460: can we use simdjson to replace rapidjson
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-1545853564
Is there any merit to use both RapidJSON and simdjson?
I think that using either RapidJSON and simdjson will reduce our maintenance cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2809862168
[This part](https://github.com/simdjson/simdjson/blob/master/doc/performance.md#free-padding), however, will require additional care:
> For performance reasons, the simdjson library requires that the JSON input contain at least simdjson::SIMDJSON_PADDING bytes at the end of the stream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "lemire (via GitHub)" <gi...@apache.org>.
lemire commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2809976587
@pitrou
> it doesn't reduce the current platform compatibility (...) we currently need to be compatible with 32-bit systems. It's not obvious that simdjson allows that.
The simdjson library is used by major systems like the Node.js JavaScript runtime, [ClickHouse](https://github.com/ClickHouse/ClickHouse), [StarRocks](https://github.com/StarRocks/starrocks), [Velox](https://velox-lib.io), and so forth. It supports 32-bit systems, big endian systems, and so forth.
Node.js code sample:
https://github.com/nodejs/node/blob/af85f3f169b5ce151ad4662cc74775e8c2973e1d/src/node_config_file.cc#L168-L188
Should you want to consider simdjson, the core simdjson will be happy to provide support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2809866592
And, by the way, we currently need to be compatible with 32-bit systems. It's not obvious that simdjson allows that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35460: [C++] can we use simdjson to replace rapidjson
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-1561548342
I'm skeptical switching to simdjson would improve performance a lot, btw. Parsing is only a small part of the work necessary to convert JSON to Arrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #35460: can we use simdjson to replace rapidjson
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-1545720496
Great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lemire commented on issue #35460: can we use simdjson to replace rapidjson
Posted by "lemire (via GitHub)" <gi...@apache.org>.
lemire commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-1545020139
We are available to help.
Note that simdjson is used by Apache Doris and ClickHouse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2795848983
Let's try simdjson for better maintainability!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "Reranko05 (via GitHub)" <gi...@apache.org>.
Reranko05 commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-5082976204
Hi! Just a note that I'm currently working on the repository-wide RapidJSON → simdjson migration through a series of smaller PRs. To avoid duplicate work, please look for another issue if you're interested in contributing. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-5411363966
@Reranko05 on main we're getting [this](https://github.com/ursacomputing/crossbow/actions/runs/32851304287/job/97812625573):
```cpp
/usr/bin/ld: /build/arrow-install/lib/libparquet.so.2600.0.0: undefined reference to `arrow::json::JsonWriter::String(std::basic_string_view<char, std::char_traits<char> >)'
/usr/bin/ld: /build/arrow-install/lib/libparquet.so.2600.0.0: undefined reference to `arrow::json::JsonWriter::GetString() const'
```
Is this something one of these issues will address or should we open a new one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "lemire (via GitHub)" <gi...@apache.org>.
lemire commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2810028915
@pitrou See PR https://github.com/simdjson/simdjson/pull/2365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "lemire (via GitHub)" <gi...@apache.org>.
lemire commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2809987618
@pitrou Point taken. You are correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "lemire (via GitHub)" <gi...@apache.org>.
lemire commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2797380908
> I seem to remember some past issues with homebrew packaging with respect to buildtime vs. runtime SIMD support (although I imagine simdjson has considered this as well!).
The simdjson library uses runtime dispatching. It is currently used by Node.js, ClickHouse and other important systems. In turn, Node.js is part of several important systems.
Optionally, we also support fancy C++20 features, and we will be integrating C++26 features (e.g., static reflection and the like) as soon as possible.
Suggested reference:
- [On-Demand JSON: A Better Way to Parse Documents?](https://arxiv.org/abs/2312.17149) Software: Practice and Experience 54 (6), 2024
Documentation: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-5411381537
> Is this something one of these issues will address or should we open a new one?
There's a PR open at https://github.com/apache/arrow/pull/50900 but I don't think it's a good solution - see comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #35460: can we use simdjson to replace rapidjson
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-1537086573
If simdjson is faster than RapidJSON for our use case too, I'm OK with this.
Could you try this and share our benchmark result?
https://github.com/apache/arrow/blob/main/cpp/src/arrow/json/parser_benchmark.cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35460: [C++] can we use simdjson to replace rapidjson
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-1561546149
Agreed with @kou , we probably want to avoid depending on two different JSON libraries.
Interested people should try working on a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2795796444
I just want to revive this discussion again. https://github.com/apache/arrow/pull/45459 will make `RapidJson` as a required dependency to parquet. However, RapidJson has been poorly maintained since 2016: https://github.com/Tencent/rapidjson/issues/2321. It also seems to be a block to support CMake 4: https://github.com/apache/arrow/issues/45985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-2809985297
> The simdjson library is used by major systems like the Node.js JavaScript runtime, [ClickHouse](https://github.com/ClickHouse/ClickHouse), [StarRocks](https://github.com/StarRocks/starrocks), [Velox](https://velox-lib.io), and so forth. It supports 32-bit systems, big endian systems, and so forth.
I see, thanks. You might want to document platform support, because [https://github.com/simdjson/simdjson/blob/master/doc/basics.md#requirements] seems a bit more restrictive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] can we use simdjson to replace rapidjson [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #35460:
URL: https://github.com/apache/arrow/issues/35460#issuecomment-3779425652
I'm sketching out how this [could look like](https://github.com/rok/arrow/pull/47).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org